PROGRESSIVE GROWING OF GANS FOR IMPROVED QUALITY, STABILITY, AND VARIATION
We describe a new training methodology for generative adversarial networks. The key idea is to grow both the generator and discriminator progressively: starting from a low resolution, we add new layers that model increasingly fine details as training progresses. This both speeds the training up and greatly stabilizes it, allowing us to produce images of unprecedented quality, e.g., CELEBA images at 1024 2 . We also propose a simple way to increase the variation in generated images, and achieve a record inception score of 8.80 in unsupervised CIFAR10. Additionally, we describe several implementation details that are important for discouraging unhealthy competition between the generator and discriminator. Finally, we suggest a new metric for evaluating GAN results, both in terms of image quality and variation. As an additional contribution, we construct a higher-quality version of the CELEBA dataset.
This paper proposes a number of ideas for improving GANs for image generation, highlighting in particular a curriculum learning strategy to progressively increase the resolution of the generated images, resulting in GAN generators capable of producing samples with unprecedented resolution and visual fidelity.
Pros:
The paper is well-written and the results speak for themselves! Qualitatively they’re an impressive and significant improvement over previous results from GANs and other generative models.  The latent space interpolations shown in the video (especially on CelebA-HQ) demonstrate that the generator can smoothly transition between modes and convince me that it isn’t simply memorizing the training data. (Though I think this issue could be addressed a bit better -- see below.) Though quantification of GAN performance is difficult and rapidly evolving, there is a lot of quantitative analysis all pointing to significant improvements over previous methods.
A number of new tricks are proposed, with the ablation study (tab 1 + fig 3) and learning curves (fig 4) giving insight into their effects on performance.  Though the field is moving quickly, I expect that several of these tricks will be broadly adopted in future work at least in the short to medium term.
The training code and data are released.
Cons/Suggestions:
It would be nice to see overfitting addressed and quantified in some way.  For example, the proposed SWD metric could be recomputed both for the training and for a held-out validation/test set, with the difference between the two scores measuring the degree of overfitting.  Similarly, Danihelka et al. [1] show that an independently trained Wasserstein critic (with one critic trained on G samples vs. train samples, and another trained on G samples vs. val samples) can be used to measure overfitting.  Another way to go could be to generate a large number of samples and show the nearest neighbor for a few training set samples and for a few val set samples.  Doing this in pixel space may not work well especially at the higher resolutions, but maybe a distance function in the space of some high-level hidden layer of a trained discriminator could show good semantic nearest neighbors.
The proposed SWD metric is interesting and computationally convenient, but it’s not clear to me that it’s an improvement over previous metrics like the independent Wasserstein critic proposed in [1].  In particular the use of 7x7 patches would seem to limit the metric’s ability to capture the extent to which global structure has been learned, even though the patches are extracted at multiple levels of the Laplacian pyramid.
The ablation study (tab 1 + fig 3) leaves me somewhat unsure which tricks contribute the most to the final performance improvement over previous work.  Visually, the biggest individual improvement is easily when going from (c) to (d), which adds the “Revised training parameters”, with the improvement from (a) to (b) which adds the highlighted progressive training schedule appearing relatively minor in comparison.  However, I realize the former large improvement is due to the arbitrary ordering of the additions in the ablation study, with the small minibatch addition in (c) crippling results on its own.  Ablation studies with large numbers of tweaks are always difficult and this one is welcome and useful despite the ambiguity.
On a related note, it would be nice if there were more details on the “revised training hyperparameters” improvement ((d) in the ablation study) -- which training hyperparameters are adjusted, and how?
“LAPGAN” (Denton et al., 2015) should be cited as related work: LAPGAN’s idea of using a separate generator/discriminator at each level of a Laplacian pyramid conditioned on the previous level is quite relevant to the progressive training idea proposed here.  Currently the paper is only incorrectly cited as “DCGAN” in a results table -- this should be fixed as well.
Overall, this is a well-written paper with striking results and a solid effort to analyze, ablate, and quantify the effect of each of the many new techniques proposed. It’s likely that the paper will have a lot of impact on future GAN work.
[1] Danihelka et al., “Comparison of Maximum Likelihood and GAN-based training of Real NVPs” https://arxiv.org/abs/1705.05263