Whoever has this problem, should send their “boys” out of the house and invite the girls in. Really, in all honesty where should they go? Is there any kind of recreational place girls can group together and just be themselves? Boys are home in the summer and not necessarily in yeshiva so should they congregate in someone’s house if their brothers are there?
Seriously if you think this is an issue get together with the other ladies in your shul and work things out for the “boys” not to be at home when you take turns to make your home “girls only” and give them some privacy.