Oversexualization of crossdressing

Why is it perfectly fine for a girl to wear her boyfriend’s baggy sweater, but does the idea of a guy wearing a tight-fitting dress shock a lot of people? This question seems to pop up a lot when talking about crossdressing and, in my opinion, is quite an interesting one. I don’t know the answer myself, but being a crossdresser and being a member of several online communities regarding crossdressing, I do have some possible explanations.