Yemeite Jews are certainly not Sephardic and never were. And other Mizrachi Jews are, also, not all Sephardic. Before the Spanish Expulsion of 1492 there were communities of Jews living in the Mizrachi/MidEast countries. They were not Sephardim, though after the Sephardim leaving Spain moved there the communities integrated in large part.
During the first emigration of Yemenite Jews, they had to choose whether to join the existing Ashkenazi or Sephardi communities and they had more in common with the Sephardi community.
Due to the Arab trade routes, the Jews in Iberia, North Africa, and Southwest Asia had enough contact to keep many of their practices in sync.