As I commented earlier one should no judge snother person by his clothes etc.
However, the following has come to mind
A man with a black hat, beard, black suit, white shirt and tie walks into a shul to give a shiur – what would the congregation say about the man? I suppose it would go along the lines of “Rabbi…..” If a guy in chinos, shirt and blazer walks in and gives a shiur, I must confess it doesn’t carry the same weight – like it or not. We would look to this guy as just that – a guy giving a shiur. Nothing wrong with that you say. Correct. But it does add to the extra kovod of shabbos of a suited guy who goes that little bit further to how he dresses.
By the way the same applies to women in their manner of dress.
The fact that the neighbor commented means that they felt the same way.
I still maintain we should not judge but maybe the way of life is that we should not cause the positionn where the comment is then made.