The correlation seems to be with trusted, male role-models that oversee a lot of young boys. Coaches, for example, have also had problems with far less media interest.
Catholic nuns are also celibate, but don’t have this issue. Religious men who aren’t able to find a marriage partners end up being celibate, but don’t have this issue. It’s an immature and overly simplistic understanding of psychology to think that going too long without relations is going to cause a man to be attracted to children. These guys are mentally ill. There is no evidence that ending the celibacy requirement would cure them and make them attracted to women instead of boys.
However, I do agree that it doesn’t seem to be an issue in yeshivas. If it were found that there were a cover-up scandal at a yeshiva, I wouldn’t really feel bad for them being sued over it.