Yes, you are understanding me wrong. I was talking about quantity, not exclusivity.
As far as yeshivish l’vush labeling people, well that’s the point (and this has nothing to do with location.) The Rabbonim and Talmidim davka want to be labeled as a Ben Torah. That’s why Bnei Torah always dressed as Bnei Torah. The precise uniform of Bnei Torah changed over the years, but Bnei Torah certainly always dressed to show they are Bnei Torah.