Most of us do not spend most of our time davening and need to be reasonably comfortable as we pursue what we hope will be solid accomplishments.
But you are right: Those who cannot maintain a dignified appearance at all times must never represent themselves as full time Torah learners. From early on I knew I could never maintain a dignified appearance at all times. I went into a profession where being sharply dressed is not important.
For the short time each day that we daven, all of us must maintain dress standards that vary from community to community.