Sam2- Because the Rema says it is.
That said, the Mishna Berura is of the opinion that we should abolish this minhag, and perhaps if it is only one garment and the gender is clear we shouldnt protest. (read: we shouldnt burn garbage cans, throw rocks, or pour bleach on them.)