Pashuteh Yid
Hashem cursed the children of Cham. Hashem blessed the children of Avraham. Oil and water don’t mix. Eliezer was a Tzaddik and him and Avraham were very close even though Eliezer was his slave. The only reason Avraham said no was because Hashem said that Elizer being a descendant of Canaan and Cham is cursed. If there was any other reason Avraham would have said it. Maybe Avraham did say it won’t work out at first and Eliezer pressured him for the reason and so he forced Avraham to explain.