I heard one frum Jew explain the wearing of black and white as representing the colours of the ink and vellum of the Torah scroll. I wasn’t raised in a religious household, so I know very little about this.
However, when I hear a knowledgeable person or rabbi teaching Torah and he’s not wearing a black hat, or kippah, I sometimes feel as if something is off.