The reason the Chachomim required cholov Yisroel was not because of inadvertent mixing. It was specifically because of concerns of intentional mixing of non kosher milk.
The reasoning for R Moshe’s heter makes sense for intentional mixing, but not so much so for inadvertent mixing.