Hi feivel.
I totally understand your point and sympathize with you.
It is my understanding that with loshon hafa one should not feel like they should just not talk. On the contrary, there are times when you have to say something and cannot stand by. But certain conditions have to be met.
Bottom line, one has to learn the laws of loshon hara. It is very important.
I hope the moderators allow this link in my post. I recommend you save it and read it each day as a new lesson automatically appears.
In a short while, you will feel like you have some grasp on how to handle matters of speech. It is a start, and that is the best place to being.