There are many Jewish employees who do go to the Office party because in this day and age they are not celebrating a religious tradition at the office party, they are just partying for the sake of partying. And yes, they might exchange gifts, but usually it is just the boss who hands out the year’s bonuses. Again it is called a Xmas party but it has nothing to do with the holiday or the religion. If the Jewish employees show up, the organizer will make sure there is kosher food available to their standard. So if a boss expects you to be at “his” party, he is usually angry and takes offense by your absence and an employee should understand that. Just as if a Jewish boss asks a non Jew to be at his Chanuka party. If that is where the boss plans to be magnanimous and give out the gifts, you can’t slight him.
This does not excuse him for his anti-semetic comments and if he said them he probably means them on some level. However, if you have a secure job at this point, I don’t think you are doing yourself or your family or the people that benefit from your maiser money a favor by rocking the boat. Like others have said, you might be the one that teaches this man a thing or two about Jews that will alter his opinion and give him a reason to respect us and not dis us.