CTL, I am with you on the suits. Gemora suggests dressing in black when going away to hide avonot …
You don’t have to look this way at inappropriate invites. Maybe chasidim do not see you as a potential chatan based on your suit, while the others care enough to ask. They may not know that you are not at that page and maybe they could use better approaches, of course. A bigger problem is when nobody asks or invites at all.