The reason why people care is because “no man is an island” People mix with each other and Gebroach eaters and non-gebroach eaters will usually mix in some form either a family event. Shopping at the same store etc.
You are forced to pay more as the store for example needs to only stock non-gebraoch items which can cost more making it more difficult for people who dont observe to celebrate Peseach