I once heard a drasha from R’ Yechiel Perr, Rosh Yeshiva of Yeshiva of Far Rockaway who explained that wearing a talis in a visible way in the street might have caused an anti-Semitic response. I think he quoted R’ Yaakov that it was proper no to wear the talis openly.
It’s a מגן אברהם in סי’ כ”ה, also brought in משנה ברורה.