Skin color, who your parents are, your religious upbringing…these are all things beyond ones control.
Kavod habriyos is very powerful in halacha. We say that causing someone emotional pain to the extent that their face becomes white is akin to murdering him. Though ironically, in such a case, Joseph then might consider the person marriage material- when their face turns white.