This is from the king of ‘not literal’:
Halevai so you are saying the avos didnt keep the torah. They can meditate with sticks fun heint biz morgen, thats not wearing tefilin.
Keeping “secrets of the torah” is beautiful but meaningless (to us).
They either wore tefilin or didn’t, and sticks aren’t tefilin.
Hence, your issue is that you are actually extremely literal.
As I’ve written on the aforementioned thread, the Medrash was given to study not to dismiss. Yes, we all know by now that Aggados are very deep and aren’t necessarily meant to take at face value. This is step one. We see a strange Maamar and we understand that it is something deeper than what meets the eye.
That is what we are taught when we start out. There is, however, a level two. We go the next step and actually try to glean an understanding. Only a fool would insist on level one when faced with level two.
Using the same neutralizing ‘not literal’ spray on every Maamar Chazal doesn’t really get you far. The idea is to learn what it does mean. For this there are Sefarim — of many types with different approaches and to varying depths.