It is not impossible if we are talking about money owedd to his mother. And, as I said, Tosafos is describing the issue of being Machlif a Nolad Bikedusha with an Horaso Bikedusha.
When Tosafos mentioned the Ichlufi (like Rashi, BTW) he was referring to any such person. Thereafter, the him and his are about the Ichlufites and not the Yoresh.
Another scenario is if the father requested that you pay it to that son but he didn’t do a Matnas Sh’chiv Mera. But, in that case Tosafos wouldn’t be running to Yibum. It would be Gezel to give it to him instead of his brother. What if the brother is Mochel the Chov? (In general, what happens when one Yoresh is Mochel?)