I think I have been running into the same problem in trying to implement WGAN-GP (never tried WGAN or LSGAN). I’ve had success on my own data with what the author calls ‘GoodGenerator’ and ‘GoodDiscriminator’ on his github 17 (I took from the nice Pytorch implementation here 17). You should definitely try those networks if you haven’t already.
I got decent results but after adding a few more residual blocks I am running into the instability you describe. I don’t really pay much attention to -D(G(z)) aka loss_G, but I do watch D(G(z)) - D(x) and I see behavior like you’re describing- D is doing fine, then a crash. D seems to recover its margin but after that the samples I get from G let me know the thing is broken.
In a smaller pair of networks, I once cured this problem by totally removing normalization from D, but maybe that was a fluke just because it was a small network.
The thing I am most curious about is which network is screwing up first, and next: why?.
I’ll be interested to know if you figure anything out about this instability, and if your generator makes any good samples after the breakdown.