Despite recent progress, Generative Adversarial Networks (GANs) still suffer from training instability, requiring careful consideration of architecture design choices and hyper-parameter tuning. The reason for this fragile training behaviour is partially due to the discriminator performing well very quickly; its loss converges to zero, providing no reliable backpropagation signal to the generator. In this work we introduce a new technique -progressive augmentation of GANs (PA-GAN) -that helps to overcome this fundamental limitation and improve the overall stability of GAN training. The key idea is to gradually increase the task difficulty of the discriminator by progressively augmenting its input space, thus enabling continuous learning of the generator. We show that the proposed progressive augmentation preserves the original GAN objective, does not bias the optimality of the discriminator and encourages the healthy competition between the generator and discriminator, leading to a better-performing generator. We experimentally demonstrate the effectiveness of the proposed approach on multiple benchmarks (MNIST, Fashion-MNIST, CIFAR10, CELEBA) for the image generation task.
This paper proposes a new trick to improve the stability of GANs. In particular the authors try to tackle the vanishing gradient problem in GANs, when the discriminator becomes to strong and is able to perfectly separate the distribution early in training, resulting in almost zero gradient for the generator. The authors propose to increase the difficulty of the task during training to avoid the discriminator to become too strong.
The paper is quite well written and clear. However there is several unsupported claims (see below).
A lot of work has been proposed to regularize the discriminator, it's not clear how different this approach is to adding noise to the input or adding dropout to the discriminator.
Pros:
- The experimental section is quite thorough and the results seem overall good.
- The paper is quite clear.
Cons:
- There is a major mistake in the derivation of the proposed method. In eq. (6) & (7), (c) is not an equivalence, minimizing the KL divergence is not the same as minimizing the Jensen-Shannon divergence. The only thing we have is that: KL(p||q) = 0 <=> JSD(p||q) = 0 <=> p=q . The same kind of mistake is made for (d). Note that the KL-divergence can also be approximated with a GAN see [1]. Since the equivalence between (6) and (7) doesn't hold, the equation (11) doesn't hold either.
- The authors say that the discriminator can detect the class of a sample by using checksum, the checksum is quite easy for a neural networks to learn so I don't really see how the method proposed actually increase the difficulty of the task for the discriminator. Especially if the last layer of the discriminator learns to perform a checksum, and the discriminator architecture has residual connections, then it should be straight-forward for the discriminator to solve the new task given it can already solve the previous task. So I'm not sure the method would still works if we use ResNet architecture for the discriminator.
- I believe the approach is really similar to adding noise to the input. I think the method should be compared to this kind of baseline. Indeed the method seems almost equivalent to resetting some of the weights of the first layer of the discriminator when the discriminator becomes too strong, so I think it should also be compared to other regularization such as dropout noise on the discriminator.
- The authors claim that their method doesn't "just memorize the true data distribution". It's not clear to me why this should be the case and this is neither shown theoretically or empirically. I encourage the author to think about some way to support this claim. 
- The authors states that "adding high-dimensional noise introduces significant variance in the parameter estimation, which slows down training", can the author give some references to support that statement ?
- According to the author: "Regularizing the discriminator with the gradient penalty depends on the model distribution, which changes during training and thus results in increased runtime". While I agree that computing the gradient penalty slightly increase the runtime because we need to compute some second order derivatives, I don't see how these increase of runtime is due to change in the model distribution. The authors should clarify what they mean.
Others:
- It would be very interesting to study when does the level number increase and what happens when it increase ? Also what is the final number of level at the end of training ?
Conclusion:
The idea has some major flaws that need to be fixed. I believe the idea has similar effect to adding dropout on the first layer of the discriminator. I don't think the paper should be accepted unless those major concerns are resolved.
References:
[1] Nowozin, S., Cseke, B., & Tomioka, R. (2016). f-gan: Training generative neural samplers using variational divergence minimization. NIPS