R’ Itzik
You could never do damage, even supposing that everyone knows who you are. No matter how far you might choose to travel away from Yiddishkeit, anytime you return you are 100% full fledged member of Am Yisrael. As you said, there are plenty of examples of this.
Have you ever considered finding a new setting? ‘Change the place, change the mazal’. If you are uncomfortable where you are, perhaps there is a different place with more people who ARE like you.