Pashut pshat in the gemara in brachos is definitely that the peope were bad. I think this is forced by the premise that we dont kick people out of the bais medrash for not reaching the highest of the high. Perhaps you would try to say that that is exactly why reb elazar disagreed but i think that makes it harder to defend rabban gamliel who was a very smart man and worthy of our defense.
As for the halevai thing, that i brought just as a rayah to the power of torah