Someone who wears a hat will identify himself as part of the group that wears that hat style, at least in the eyes of people who look at such things. It really makes no difference in the great scheme of things, but gives people something to shmooze about in various real and online coffee rooms so what harm can it do?
PS
FWIW, when I go to Rav Hershel Schachter’s shiur in flatbush on Monday nights there is a maariv minyan afterwards. Not once has he asked the baal tefila to put on a hat.