The problem is that there is more than sufficient reason to believe that these collectors are NOT the ones who live in poverty in Mea Shearim or even Mattersdorf, but rather those who have a few bank accounts hidden away, or even those who take what they collect and head off to Atlantic City.
Those who truly live in poverty are modest people who know not to disturb davening. They are after all too modest to approach organizations that can help them make a proper Shabbos every week, and we know that Y-m is full of such organizations.
Those who collect on behalf of those who do need it often collect in an organized manner.
The answer is some sort of authorization system – and also an end to the driver percentage scam that ensnares the honest collectors as well as encouraging the crooks.