ujm,
“And only put on special clothing for davening and Shabbos. “
But if someone comes to davening in jeans, T-shirt, and baseball cap, there is a high probability that the person does not self-identify as chareidi. So yes I agree with you that clothing is a secondary point and is not integral to what’s most important about being chareidi – as the expression goes, the clothes don’t make the man. But the clothes do tell us what the man wants us to think about him upon our first impression.