How can I get the iPhone Podcast app to resume playback after extended stop? I noticed a while ago that the music app didn't play podcasts any more and that a (new?) podcast app does that.

The result is that when I now listen to a podcast and stop it with the button on the headphones and then try to restart it some time later, the iPhone plays music from the music app instead.

In the old days when podcasts were in the music app, the podcast would continue playing.

How can I get the old behaviour back, i.e. I want the iPhone to continue playing what it was playing when I last stopped it?
The behavior you describe is inconsistent. In my experience, sometimes the podcast will continue and sometimes the iOS device will switch to the music app. I believe (but haven't confirmed) that this depends on whether or not the podcast app, be it Apple's or third party, is still considered active by the system. When it occurs I just open my podcast app and playback of the the last podcast resumes (unless I was streaming it in which case I may have to hit play.)

Because this is a memory management issue there is no way to force the behavior you seek using a dedicated podcast app. However, if you uninstall Apple's Podcasts you can continue to download podcasts using iTunes and can move them over to your iOS device with a playlist. This solution is less than ideal—for example you can't download fresh podcasts directly to the device. 

iOS 7 UPDATE: It is my understanding that under iOS 7 you no longer have the ability to listen to podcasts from within the Music app. Perhaps better memory management and multitasking will provide resumed playback more consistently, but it seems that a podcast client (or listening via Safari) will be required.