I feel like the only thing you can do is be upfront with him. There's no way to sneak out of it, and you're not really obligated to say for the sake of his happiness.  Your happiness or his?

I'd recommend next time to be explicitly clear that you only want something casual. That way, it'll feel easier leaving, and they won't end up so attached