DY: Sadly, there are far, far too many people who think just like that.
I remember once attending a Shiur at a local Shul. The Shiur was given by one of the Balabatim. There happened to have been some representatives of the Agudah visiting this Shul (I don’t remember precisely why; I think they were visiting every Shul that has a Daf Yomi or something to be Mechazek them before the Siyum or something like that) asked the person if he wanted them to give the Shiur instead. He said no thank you and went on to present a brilliant bit of Lomdus that was exciting and still understandable for the Baalei Batim. One of the representatives when up to him afterwards and said he’s the first Srugi he’s ever met that’s also a Talmid Chacham.
Or, to quote the Marvelous Middos Machine: “There’s that boy with the funny-looking knitted Yarmulke… Why would he need a Gemara?”