Here is the quote (Soncino translation) from the Gemara Pesahim 49A:
For R. Johanan said: If the daughter of a priest [marries] an Israelite, their union will not be auspicious. What is it? Said R. Hisda: [She will be] either a widow or a divorced woman, or she will have no seed [children].
In a Baraitha it was taught: He will bury her or she will bury him, or she will reduce him to poverty. But that is not so, for R. Johanan said: he who desires to become wealthy, let him cleave to the seed of Aaron, [for it is all the more]
From what I remember from Rav Ovadia, the problem is only if a Bat Cohen marries an Am-Haaretz. If he’s a regular frum baal habayit – e.g. he works and regularly learns – then no problem. All the better if he’s more learned.