Whynotme, whatever the truth is about Chofetz Chaim, you have to be happy where you are. It sounds like you aren’t happy there, and perhaps you will learn better where you’re happier. Rabbi Dovid Harris Shlit”a has his kids in Lakewood, so clearly he believes there’s learning outside of Chofetz Chaim.
