The Baron is slow to pull away, but does so.  "I do hope to see you make more of an effort to succeed in this dungeon from now on, my dear.  I may interpret future failures as a desire to enter my harem permanently."  Having said that, Baron Sukumvit vanishes again, leaving you to compose yourself and continue on.