cshapiro –
Yeah, I get the picture.
It sounds like it’s something that’s not going away, and silence might not work here. So talk to him. I think the best way to say it is to make it about you. Keep in mind that people don’t enjoy when you put down their friends. So don’t try to convince him with example upon example about how much of a moron his friend is, even if he is. Say that you don’t feel comfortable because your personality clashes with his and his wife’s, etc. And be supportive of him, give alternative ideas for who to hang out with, and so on. Good luck.