Phil:
Let’s say that hypothetically, one of Rashi’s children or grandchildren specified in writing that Rashi did not mean it literally, and that he meant it exactly as you are presenting it, but–for whatever reason–the greater Orthodox world ignored this and continued to interpret this as we’re saying. This hypothetical situation would not change ANYTHING about what I’m saying to you.
In our current reality, you are a daas yochid on this. That might mean that you’re right and that the rest of the frum world is wrong. But, often times in life, it’s better to be normal than to be right. I would maybe suggest you look up the english definition of the word “orthodoxy” if this style of thinking seems crazy to you.