For six hours, every single time? It is a stretch.
It’s not a stretch. Also, consider that even karov levadai is considered vadai in every place I’ve seen.
It’s still not always/vadai. Besides, following Rashi and the Rambam lechumra is not the ikkar hadin either. The Tur’s lashon is ??? ????? ?????? ??? ?????.
Some reading comprehension- Following Rashi or Rambam “LeChumra”, only applies where they argue, such as after 6 hours if there is still meat in the teeth, according to Rashi assur, Rambam, muttar. Or if one chews on meat for a baby, Rambam assurs, Rashi matirs. These arguments are what the Tur suggests that you should always be machmir. Now: If a person ate a piece of steak, Min Hadin, according to both shittos, it is assur to eat dairy within 6 hours.
It is also clear from the Taz that the Rambam, (Rashi) and Rif hold that it is assur midina, not just a chumra to eat dairy within 6 hours. Additionally, he notes, that even those that are meikil and hold one hour, (ie. obviously not bnei torah), only can rely on the Tosfos if one does Kinuach and Hadacha. (I understand that the kinuach and hadacha would downgrade the issur according to the Rambam from a issur vadai, to a chumra because of a chashash, while the mekilim hold that the one doesn’t have to be choshesh for the various possibilities.)
Then you also have the Pri Megadim, who holds that even if you have a scenario where according to both Rambam and Rashi it would be muttar to eat dairy, still he holds that during the 6 hours, lo plug rabanan and ein lifrotz geder.
The minhag is in accordance with what the Rambam held lehalacha, but it is a minhag, not the ikkar hadin as the Rambam himself held; as I explained the Rema.
Ayen the Taz 2 and Shach 9, who obviously hold that it’s not just a minhag, as minhagim are meant for everyone, not just bnei torah. Nor is it that in every argument where there are mekilim does it make the more stringent shitta just a chumro.