I don’t think it’s a Midrash that says she was 3, I think it’s Rashi. There’s a Sifrei that says she was 14.
Rashi’s reasoning is that her birth is mentioned immediately after the Akeidah, so he implies it happened at the same time – as is the case with the death of Sarah. Yitzchak was 37, and Sarah was 127. Since Yitzchak married at 40, Rashi says Rivkah was 3.
The Ibn Ezra writes that Yitzchak was 13 at the akeidah, which means it would be another 24 years until Sarah died. In that case, all the events mentioned there didn’t occur at the same time, so there is no way of knowing how old Rivkah was at her wedding.