Feif,
Just so that you understand their point of view.
Because this Rav that was invited to speak represents something that those people (the non attenders) are against. They felt that it would be inappropriate for them to attend.
To those people, it was L’havdil as if an apikoress would be invited to speak. Although they badly wanted to attend, they felt that according to their beliefs it would be inappropriate to do so.