The difference between a legitimate Halachic opinion and holding by a very minor Daas Yachid is that it is wrong. So yes, if Chalav Stam was an incredibly minority opinion that no one held like, then it would be wrong to force people to hold like it and probably Assur to do so. Since it’s a relatively mainstream opinion, that’s fine.
This is not Davening, or Tznius, or Chalav Stam. This is an incredibly Mechudash Shittah that doesn’t make any sense in practical Halachic terms. R’ Vosner was entitled to an opinion. Random Rabbonim of communities are not, especially in an area such as this.