No hard feelings, but that’s not aggada.
It not not merely some vague message about the value of learning incorrectly using the lomed as an analogy.
It’s not literal in the sense that it’s a blanket issur, or that there aren’t factors to consider, but it’s a real preference based on what is typical for a talmid chochom. So sure, if what the gemara wants can be found where it’s not literally a talmid chochom (e.g. the Chazon Ish saying that marrying a Bais Yaakov girl is a kiyum of marrying a bas talmid chochom) that’s also fine.
The message that the gemara is saying isn’t vaguely about Torah, it’s about marriage, and therefore brought in those places in Rambam/Shulchan Aruch. The fact that the gemara expressed it in terms of a talmid chochom is because that is typically where the qualities desired are found.