Incorrect. That is not a problem with a chumra at all. That is a problem with the person in question.
Additionally, cholov yisroel isn’t a chumro altogether. Cholov yisroel is required per Chazal. Non-CY is a kula.
Perhaps we should have a thread about the problems with kulos.