I think this is my 3rd time quoting the ohr somayach, but it’s a very, very useful geder of bitul torah. He says it grows with you, that it depends on where you’re holding in your capabilities, and that Hashem does not give us mitzvos that we are sometimes unable to do…the minimum is krias shema in the morning and night, but the ceiling never ends.
I agree that mechanchim shouldn’t dismiss anything people do besides learning as bitul torah. There are also usually much more compelling problems in the aforementioned list of activities that are not subject to the relativity of bitul torah.
It is noteworthy though that the ohr somayach is a bit of a chidush and that many rishonim seem to take for granted that one is obligated every second to learn barring necessities such as parnosa, eating, sleeping, etc
That being said, bitul torah is no joke; the zohar says that there are 7 circles of gehinnom, each one is comprised of all the others, and the 7th, comprising 49, is for those who could have learned but didn’t. “Efshar lo lilmod veayno lomeid” it’s also one of the 3 people that Hashem “cries” over.