There’s a gemara in kesubos that says a person is only insulted if it’s true
There’s a mishna in sotah that says a the price of boshes is all according to the one who embarrased and the one who was embarrased
so it’s probably because you don’t feel what he says is meaningful