Tosafos is specifically saying that the “big trouble” would be yibum problems.
Right. And therefore..? Yibum is where it would be a problem for people to think that this type of person is a full fledged born Yisroel. Tosafos is not applying it to this person. All these types of Gezeiros work this way: we don’t want people to draw wrong conclusions.