Truth be told
How do you think the “black hat” phenomenon developed? Do you concede it had something to do with that time period/society’s accepted forms of dress? What’s changed since then?
And the story about the fellow walking to shul does not prove anything. If someone has the custom to wear a black hat on shabbos and neglects that minhag due to the heat, then he is essentially neglecting what he has deemed kavod shabbas because of the heat. That’s not difficult to accept.