Your comparison to nazi atrocities and their perpetrators is misguided and rather shameful. Slavery until the US had a Civil War about it was global, especially in Africa. It was such an old institution it is even in the Torah! Does it make sense to tear down statues of our forefathers because they had slaves?
I’m obviously not justifying slavery, but the people who contributed to our society are not negated because they had slaves. They would, however, be negated if they perpetrated the murder of six million men women and children.
You’re also ignoring the fact that something that bothers you is not the same as assuming the right to destroy it, but a previous post already addressed that…