Squeak, I read your whole post, but you know that deliberately approximating the molad and the value of pi or the square root of 2 to simplify calculations, which I completely accept, does not seem to be comparable to what the Rishonim were discussing in Bava Basra. Tosfos seems to say that the diagonal of a 4×6 rectangle is less than that of a 5×5. If he wanted to approximate, why not agree with the Rashbam who, one could for arguments sake, say that he chose a 5×5 as an approximation to a 4×6, since it is a nice round number.
But Tosfos argues and says that the Rashbam is wrong, and that the diagonal of a 4×6 is actually less than a 5×5, since a 5×5 has 25 [square] amos, while a 4×6 has 24 [square] amos. But, in reality, the 4×6 has a longer diagonal. So while you could in theory defend the Rashbam with your approach (gemara sometimes says lo dak), however, when Tosfos comes to differ and complain that the Rashbam’s value is not exact enough, and instead gives a different value which it turns out is even more inexact and in the wrong direction, then there is a problem. In addition, we both know that using the area of a rectangle is not a good way of gauging the diagonal, since for example a 1×12 rectangle would have a diagonal more than twice that of a 5×5, even though its area is less than half.