Disclaimer:
I am also fairly new to GANs, but I've been extensively playing with things, and trying various ideas to get something usable (I'm also using PyTorch). So I am by no means an expert, but after seeing your question, I thought I'd share some things I've learned along the way with the hope that you'd find them useful. I haven't looked thoroughly at your code yet, so I assume your code is generally correct (meaning no unwanted silly mistakes with the network models, loss computation, etc...).
Also, beware that I'm not using MNIST and my architecture has recurrent layers. So not all these suggestions might be applicable to you and YMMV...
Q 1. Am I seeing mode collapse?
After some tweaking and iteration I have a GAN which does learn to generate images which look like they might come from the MNIST dataset. Actually they're not digits yet but they are recognisable pen strokes, and certainly not random noise.
Remember that mode collapse happens when your network fails to generate a diverse enough set of outputs (most/all samples look the same).
Looking at your example image, I don't think you're even at the point of mode collapse yet. Your example image does not look much like a "real" digit. In my experience, when mode collapse happens, your generator would produce valid and almost convincing looking example, but that's it. All generated examples will look pretty much the same.
That being said, I think your problem is that your generator network has not yet learned to produce real (or semi real) looking samples. I'd suggest training for longer, while making sure that your loss computation, etc. are all correct. When my network started working, I could immediately tell that it's producing valid output.
A few pointers:
* Be sure that your learning rate is small enough. 740
My first problem was a large learning rate (I used 0.001 with <PERSON>, then realized that my model only works with something small like 0.0002).
* Make sure learning is happening. Track the loss values over time and make sure they make sense. There shouldn't be any spikes in loss values, else something is wrong.
I've had great success with feature matching as my loss metric. Without feature matching, my network never really worked well.
Q 2. How To Escape Mode-Collapse?
That's the million dollar question! I've been pulling my hair out trying to find a working solution for mode collapse for the past month.
You see, my problem right now is that all generated samples look super convincing, but they all look almost exactly the same.
If the answer above is that a train GAN should output many diverse but valid images, then I have mode-collapse.
I've read extensively and tried many approaches to avoiding mode-collapse but none have worked:
with / without batch normalisation with / without maxpooling with / without dropout with / without label softening with / without noise added to both input and target labels, including noise that decays over training time various widths and depths for the generator, less so the discriminator increasing training time (but poor compute power only allows me about 6-10 epochs on the full dataset)
It's great that you tried all this (I did too), but I found that for me, no architectural change really made any difference. The only thing that has partially worked for me so far is using the WGAN loss. It's very tricky to get it to work, but after about 2000 epochs, I see samples that look real (with some obvious flaws) but look very different from one another.
I suggest you try both WGAN and WGAN-GP as your loss metric. Due to reasons beyond this discussion, I cannot easily use WGAN-GP.
Lastly:
Do they have to be matched but opposite? mine aren't - the discriminator is proven to have the learning capacity and that's it. After that the shallower the better to ensure easier back propagation to the generator.
My discriminator and generator almost match and are opposite of each other. I've tried many many variations, but didn't find any significant differences/improvements. I just decided to keep things simple and match them for now. Once my mode collapse issue goes away, I will revisit this.
Hope this helps. I really hope to get feedback from some experts on this. 458