ItcheSrulik: He would alienate the non hatters if he avoided them being the Shatz. And that would result in them possibly not attending shiur. He may feel its not worth the trade-off, even if less than ideal.
AOM: Reread my comment. The bathing suit remark was a hypothetical comparison.