I’m pretty sure it’s assur. But as always, all halacha li’maseh shailos should be asked to your LOR, not to anonymous CRers that may or may not have a clue what they’re talking about.
“For reasons of Sholom Bayis or even better hygiene – why not?”
If it’s assur, its assur regardless. If it’s muttar, there’s no need for these reasons. I’m not sure what difference these justifications should make (or, for that matter, what hygiene issue there is with chest hair).
“And if women have a mustache or chin hair, are they committing the aveira of beged ish if they likewise remove that hair?”
No, its beged isha for a man precisely because women do it. If something is assur because of beged ish/isha to one gender, by definition it’s muttar (as least beged ish(a) wise) to the other.