Mentsch, I think what was being referred to was when they had an issue with men & women mixing in the crowds. The Baba Sali himself wouldn’t go for this reason. However, they have fixed this issue & many holy tzaddikim are known to go there.
Either way, I don’t agree with the “bitul torah”. Was it bitul torah when the Arizal also went to Kever Rashbi & danced on Lag B’omer?