I’m totally with you on this. I’ve heard the complaint before, it’s been spoken about. Especially when tznius issues are at play, and the subject feels much more needless discomfort.
It’s not a real reason, but I think it occurs more by Yidden because of their inner concern, caring attitude. Again, it shouldn’t be a heter, I think that’s just where it comes from. Anyone else would have a natural MYOB attitude. It’s just a Yid’s natural instinct at play. Sometimes one has to overcome natural instincts when it causes another discomfort. (my humble opinion)