I have seen brought down (forgive me, I don’t recall the exact source.) in the name of Chasidic Rabonim that is preferable to walk through, holding something even as minor as tzitzis or peyos, then to make a big to do of asking the ladies to move or asking somebody else to walk through with you (and potentially getting them stuck on the other side or being machshil them, and walking back through)
So perhaps have a serious conversation with your Rabbi or Moreh Derech regarding what actual lessons to take from us.