&gt;How do I handle not extending that invitation politely?

Honestly, you don't. They're a couple, they come as a pair.

You may need to have a talk with him about that this isn't a sustainable friendship for you. You tell him he needs to be more respectful of your friendship, or you need to not be in it. If it's causing you too much distress, you may need to 'break up' with him basically. You can ask him to try to handle the issue more directly, but that's about it. It's up to him to manage his relationship. If he loses friends over it, that's his choice to make.

Any attempt to keep him but not her is going to exacerbate her concerns and make it worse. You can try reaching out to her and having a heart to heart about how much it sucks and that you'd like to work on resolving it and asking if she knows what might help. But if you do that, you need to be prepared for the possibility that nothing will be resolved. Many people don't want to deal with their jealousy or insecurity issues and/or don't know how to.