I agree with the OP’s sentiments.
Our purpose in the world is to become like Hashem. “Mah nu rachum, af attah rachum, mah hu…” These are all issues of bein adam lachaveiro. It does not say “mah hu tzaddik af attah tzaddik”.
The purpose of the mitzvos is that they improve our middos. Of course, we do mitzvos because Hashem commanded them, not because we are trying to improve our middos, but chazal have told us that the reason for them is that they improve our middos.
So why do some disparage bein adom lachaveiro?
I theorize it is because other groups have taken the bein adom lacheveiro’s and made that the entire torah, which it is not. By doing so they actually destroy their ability to work on their middos, since Hashem created man in a way that the torah is what can improve him.
I theorize the disparaging of bein adam lachaveiro one occasionally hears (although not very often), is a backlash to those who are denying the place of bein adom lamakom.
What do you think?