Sam,
It’s not that simple.
According to many it’s dependant on g’dolim or k’tanim.
Accordingly, many pasken that due to sofek b’rachos we don’t make the b’rachos again.
You are correct though, that it is 100% muttar to speak with t’fillin. And as usual, people are way more makpid on chumros than halachos.
One of the craziest examples was ther time I saw someone lift his shel rosh a bit to be able to speak loshon horo!!