amom,
I understand your question and an excellent suggestion.
To answer your question – everyone cares about Learning and people in kollel are our nation’s front line. A similar situation is that a machlokes about one building in Yerushalaim gets more interest in the world than a big war in Africa. So, you should be proud that people care about your family activities, whether supporting or criticizing or giving you suggestions. I don’t think you would prefer an alternative when nobody cares. As a 19th century author writes: if the Czar understood the value of learning, he would put a soldier near every Jew. [not for protection, but] if a Jew gets distracted, the solider will kick him with the bayonet.
As to work & learn, I think you are right. It seems that working professionally one day a week should be enough for humble living, leaving 5 days a week to learn. Do we know anyone like that. I am told that some Persians used to do that – they’ll travel to remote areas for several weeks, buy exotic things there, come back, sell them abroad, and then learn the rest of the year.