&gt;  the way he handles all these encounters feels extremely manipulative bordering on abusive

I'm not even sure I'd say "bordering."  Your kids love him because they don't realize that he's a hypocritical, projecting, selfish asshole whose understanding of consent is shaky at best.  You need to leave because that's not the kind of role model they need - and, of course, because you deserve so much better than this.  Get out, and work on figuring out why you're having so much trouble identifying when someone's behavior is out of line and setting boundaries so that you don't wind up in another unhealthy relationship.