User, there’s a lot to unpack in those statements. Smoking, especially casually, is widespread in the yeshiva world, and was common among roshei yeshiva until recently. One of the ziknei rosh yeshivos still smokes. it has nothing to do with going off the derech as far as i can see. At worst, it is a poor life choice and a possible violation of venishnartem meod es nafshosaychem (though continuing to smoke if one was unaware ot the danger is allowed according to most); it will also limit one’s success in shidduchim.
Being involved in olam hazeh, as you put it, doesn’t mean one is off the derech either – if someone is involved in futile pursuits that have no redeeming spiritual value, then it is against the Torah, and according to some rishonim is an issur asei of veahavta es Hashem elokecha. I would agree that here there is a connection – everyone who goes off violates this precept, but not everyone who transgresses it is off the derech.
Going off could be described as engaging in consistent behavior that is contrary to the minimal norms of Torah observance according to one’s upbringing. Meaning if an MO kid is with girls, his aveirah is of course unacceptable, but one cannot term him “off the derech”, since to him, this “lifestyle” isn’t a contradiction to shmiras hamitzvos, whereas such behavior definitely would place a mainstream student who is aware of the severity of this action in that group.
I’m not convinced that the line is clear cut for women, actually, the more I think about it, I believe it’s the opposite Women have a multitude of dress choices, none of which make a statement necessarily about their frumkeit, whereas a man who wears even a grey suit has made a statement about where he sees himself, certainly if he’s wearing colored shirts, tan, green suits etc.
Women are also encouraged to engage in hobbies and have interests. Their viewpoints are also more “open minded” and they are not held accountable for deviant hashkofos, because people don’t notice them..men are taught by rebbeim not to be medayak and address hashkofic problems in what women say on dates, because “they don’t learn gemara, so how can you expect them to have daas torah?”. Of course there is a line drawn regarding things like feminism, or castigation of rabbinic authority.