“It’s not about voicing it. It’s still not okay.”
I don’t think that’s true. I think that halachically the only problem is if it voiced. Actually, in this case, I think it would be a positive thing – those who disagree can work on trying to see those people in a better light and try to figure out why others see things that way.