WIY- I hear you , I have the same problem- yes its a problem!
I never noticed I did it until one friend put me in my place. In essence what we are saying to the speaker is that we couldn’t care less about what they are saying. (its probably a bit of ADD)