The Alshich answers that Lavan is the source of all our problems. If he would have given Rochel to Yaakov then Yosef would have been born first, and the brothers would never have fought with him. Thereby never selling him to slavery, this would have prevented our exile in Egypt from ever happening. That is why he is mentioned as he was the source of our going down to Egypt and not Eisav.
Reb Yaakov Kaminetsky says it is not just Lavan it is all Armenians. As when Yaakov was going to marry Leah and not Rochel, they all knew and said nothing.
The Brisker Rav has a different reason. The declaration of Bikkurim is meant to thank Hashem for a deliverance from hardship. Eisav is still around and still can cause us problems while Lavan is no longer around so in our declaration we are thanking Hashem for delivering us completely from Lavan but Eisav as an enemy to the Jews still stands.