Neville,
It’s every bit as shallow to judge someone who wears a white shirt as being intolerant as it is to judge someone who wears a colored shirt as being “leftern” (your word).
People have every right to notice the clothing that someone else is wearing but they’re not free to judge and stereotype the person if they’re not violating halacha. The fact that this happens in certain places is not to their credit.