The prohibition is that of a man dressing up in order to look like a woman , or vice versa. The reasons are that they would use this tactic to purposely mingle for unkosher purposes (Bach et al.). Some say (Rambam sefer hamitzvos) that there are prohibited magical connotations to this cross-dressing. There are also other issues here, see Nazir 59a and Tur/SA YD 182.
Therefore, it is permissible to wear men’s clothing when it is clear that the purpose is benign. This includes (but is not limited to) such things as staying warm or cool, unisex articles of clothing, simple articles of clothing that are not jewelry, or when the purpose does not indicate a desire to appear like the other gender. It would be permitted to wear a man’s shirt for painting the house, or if one is “running late” and just grabbed something to wear. These are clearly benign actions.
When the purpose is davka to look like the other gender, but the intent is for fun or entertainment, the opinions are divided (see Sefer Yereim, Bach, Rama). This includes cross-dressing for “shtick” at weddings and on Purim, which was common as long ago as seven-eight hundred years.
Wearing a man’s shirt for tznius would certainly be permitted, although I think that one should be able to find tznius clothes designed for women. After all, proper women’s attire designed and manufactured for women is not a new phenomenon.
I find it implausable that button position and the like can be included in this prohibition, and one can safely ignore “the button thing”.