cantoresq, I am pleased to inform you that you are wrong. I think the vast majority of the yeshivishe velt, myself included, never knew this term to refer to anything derogatory (or otherwise for that matter.) In fact, as hard as I try, I am having difficulty to recalling ANY derogatory vernacular specific for use within the yeshivishe velt.
And regarding Baale Teshuvas in general, all my life I’ve only heard the most laudatory praise for them. Certainly nothing derogatory.