Besides for what DY pointed out, if I remember correctly the MB says that a person needs to have control over his hirhurim as well.
But still, I think for the kolleleit who is sitting in a BM all day, none of this should really be a problem. I’ve tried finding out why we don’t wear them all day. The most that I’ve heard is that once it was established that we only wear them for shachris because the klal can’t keep a guf naki all day (and kavanah), so everyone stopped.
But once in a very rare while you pass a GR”Anick wearing tefillin under his hat in E”Y. I think there’s a Yeshivah in the Old City that wears tefillin all day.