anon1m0us,
The very fact that you cite examples of “black hatters” engaged in immoral behavior is an indication of what a black hat is supposed to stand for. The reason that the image conveyed is so striking is because it seems so paradoxical, specifically because the black hat is supposed to symbolize moral purity.
Of course, someone is not judged, ultimately, by his levush. But there is important symbolism and association. The fact that some, unfortunately, do not live up to what their exterior should represent is indeed a chillul Hashem, R”l, but is not a reason to disregard how one chooses to appear.