there certainly are differences between the sweaters of each gender (size, shape, pattern, neckline, etc). The issue is that since a men’s sweater is actually more tznius than a women’s, perhaps the issur doesn’t apply. The gemara says the whole point of outlawing crossdressing is to promote tznius (“kedei shelo teilech bein ha’anashim”). Is it being darshinan taima d’kra to say that if you’re not breaking tznius (and fakert, strengthening it!), then it’s mutar to wear the clothing item?