“there would be a very plausible reason why non-kollel people would want kollel people to leave kollel and not the reverse, namely that the non-kollel people are generally the ones supporting the kollel people.”
No one is forcing anyone to support anyone else, in Kollel or out of Kollel. I have no tainas if someone chooses to give their maaser to things other than Kollel and not a dime to Kollel. So don’t give. But where do they come about spouting against other’s choice to attend Kollel?