Colleges that Avigdor Miller and Lubavicher Rebbe attended probably did not have a lot of women at the time. It is a known social observations that groups with less women are more moral (i.e. women are in demand and can insist on their rules).
There is also a difference between attending a local college with kids attending classes and going home v. shipping kids to a campus.