Well it does have specific meaning. It means you’re a Jewish male. Whether or not you want that honor, it is your honor nonetheless.
I am not saying people should call you it against your will. Maybe you also don’t like being called Mr. That’s your prerogative. But you are not less of a Mr. just because you don’t want to be called it and you are not less a Reb just because you don’t want to be called that.