This is our experience in More charedi neighborhoods
Unless you are REALLY Untzniut (Meaning something like shorts and Tank Tops or something along those lines) they generally know who is charedi and who isnt , and if you arent charedi, they dont really bother you