I think you need to reframe the situation. Currently, you seem
to be thinking that you are meant to deliver a good shiur, to
yours and the participants’ satisfaction. I would suggest thinking
instead that you are there to fulfill the needs of those regulars,
to interact with them in whatever manner they will be happiest about.
(If you don’t like that suggestion, perhaps you should call the rav
and ask him how he usually handles the issues you are having.)