LARGE SCALE GAN TRAINING FOR HIGH FIDELITY NATURAL IMAGE SYNTHESIS
Despite recent progress in generative image modeling, successfully generating high-resolution, diverse samples from complex datasets such as ImageNet remains an elusive goal. To this end, we train Generative Adversarial Networks at the largest scale yet attempted, and study the instabilities specific to such scale. We find that applying orthogonal regularization to the generator renders it amenable to a simple "truncation trick," allowing fine control over the trade-off between sample fidelity and variety by reducing the variance of the Generator's input. Our modifications lead to models which set the new state of the art in class-conditional image synthesis. When trained on ImageNet at 128×128 resolution, our models (BigGANs) achieve an Inception Score (IS) of 166.5 and Fréchet Inception Distance (FID) of 7.4, improving over the previous best IS of 52.52 and FID of 18.65.
Summary:
The authors present a empirical investigation of methods for scaling GANs to complex datasets, such as ImageNet, for class-conditioned image generation. They first build and describe a strong baseline based on recently proposed techniques for GANs and push the performance on large datasets with several modifications presented sequentially, to obtain strong state-of-the-art IS/FID scores, as well as impressive visual results. The authors propose a simple truncation trick to control the fidelity/variance which is interesting on its own but cannot always scale with the architecture. The authors further propose a orthogonalization-based regularization to mitigate this problem. An investigation of training collapse at large scale is also performed; the authors investigate some regularization schemes based on gathered empirical evidence. As a result, they explore and discard Spectral Normalization of the generator as a way to prevent collapse and show that a severe tradeoff between stability and quality can be controlled when using zero-centered gradient penalties in the Discriminator. In the end, no solution that can ensure quality and stability is found, except having prohibitively large amounts of data (~300M images). Models are evaluated on the ImageNet and on this internal, bigger dataset.
Pros:
- This investigation gives a significant amount of insights on GAN stability and performance at large scales, which should be useful for anyone working with GANs on complex datasets (and that have access to great computational resources).
- Even though commonly used evaluations metrics for GANs are still not fully adequate, the authors obtain quantitative performance significantly beyond previous work, which seems indeed correlated with remarkable visual results.
- The baseline and added modifications are well presented and clearly explained. The Appendices also have great value in that regard.
Cons:
- Discussions sometimes lack depth or are absent.
For example, it is unclear to me why some larger models are not amenable to truncation. Besides visible artifacts, what does it mean? Why does a smoother G reduces those artifacts? Were samples from those networks better without using truncation? Why would this be?
Authors report how wider networks perform best, and how deeper networks degrade performance. Again, discussions are lacking, and it doesn’t seem the authors tried to understand why such behaviors were shown.
Even though this is mostly an empirical investigation, I think some more efforts should be put in understanding and explaining why some of those behaviors are shown, as I think it can bootstrap future work more easily.
- In Section 3.1 : “Across runs in Table 1, we observe that without Orthogonal Regularization, only 16% of models are amenable to truncation compared to 60% with Orthogonal Regularization.” For me, this is not particularly clear. Is this something the reader should understand from Table 1? 
- I question the choice of sections chosen to be in the main paper/appendices. I greatly appreciated the negative results reported in the main text as well as in the appendices and this has significant value. However, as this is to me mostly a detailed empirical investigation and presentation of high-performance GANs on large scales, I would be likely to share this with colleagues who want to tackle similar problems. In this case, if future readers limit themselves to the main text, I think it can have more value to present some content form Appendix B and C than to have more than a full page on stability investigations and attempted tricks that turned out not to be used to reach maximal performance. However I do not want to discourage publishing of negative results, and I definitely wish to see this investigation in the paper, but I merely question the positioning of such information. With regard to my first negative point above about the lack of discussions, it seems the analysis of Section 4 is disproportionate compared to other places.
Suggestions/Comments:
- Regarding the diversity/fidelity tradeoff using different truncation thresholds, I think constraining the norm of the sampled noise vectors to the exact threshold value (by projecting the samples on the 0-centered hyper-sphere of radius = threshold) could yield even more interesting or more informative Figures, as obtained scores or samples on the edge of that hyper-sphere might provide information on the ‘guaranteed’ (not proven) quality/fidelity of samples mapped from inside that hyper-sphere. 
- In Appendix D, the Figures could be slightly clarified by using a colored heatmap to color the curve, with colors corresponding to the threshold values. Similar curves could also be produced with the hyper-sphere projection proposed above to have a slightly clearer idea of the behavior on the limit of that hyper-sphere.
- In Section 4.2, in the second paragraph, you refer to Appendix F and describe “sharp upward jump at collapse” in D’s loss. However, it seems the only Figure showing D’s loss when unconstrained is Figure 26, in which it is hard to notice any significant jump in the loss.
- In Appendix F, Figure 20 d), the title seems wrong. It seems to report sigma^2 values, but the title says “losses”.
This investigation of GAN scalability is successful results-wise even though the inability to stabilize training without sacrificing great performance on ImageNet is disappointing. The improvement over previous SOTA is definitely significant. This work thus shows a modern GAN architecture for complex datasets that could be a strong basis for future work. However, I think the paper could and should be improved with some more detailed analysis and discussions of exhibited behaviors in order to further guide and encourage future work. It could also be clarified on some aspects, and potentially re-structured a bit to be better align with its probable impact directions.  I would also be curious to see the proposed techniques applied on simpler datasets. Can this be useful for someone having less compute power and working on something similar to CelebA?