Generative adversarial networks (GANs) are a class of deep generative models which aim to learn a target distribution in an unsupervised fashion. While they were successfully applied to many problems, training a GAN is a notoriously challenging task and requires a significant amount of hyperparameter tuning, neural architecture engineering, and a non-trivial amount of "tricks". The success in many practical applications coupled with the lack of a measure to quantify the failure modes of GANs resulted in a plethora of proposed losses, regularization and normalization schemes, and neural architectures. In this work we take a sober view of the current state of GANs from a practical perspective. We reproduce the current state of the art and go beyond fairly exploring the GAN landscape. We discuss common pitfalls and reproducibility issues, open-source our code on Github, and provide pre-trained models on TensorFlow Hub.
This paper seems to be an exposition on the primary performance affecting aspects of generative adversarial networks (GANs).  This can possibly affect our understanding of GANs, helping practitioners get the most in their applications, and perhaps leading to innovations that positively affect GAN performance.
Normally, expositions such as this I find difficult to recommend for publication. In these times, one can find "best practices" with a reasonable amount of rigor on data science blogs and such. An exposition that I would recommend for publication, would need to exhibit a high sense of depth and rigor for me to deem it publication worthy. This paper, for me, achieves this level of quality.
The authors start off by giving a precise, constrained list of hyperparameters and architectural components that they would explore. This is listed in the title and explained in detail in the beginning of the paper. The authors are right in explaining that they could not cover all hyperparameters and chose what I feel are quite salient ones. My one ask would have been a survey of how activations might affect performance. I sense that everyone has settled upon LeakyReLUs for internal layers, but a survey of that work and experimentation within the authors' framework would have been nice.
The authors then explain the metrics for evaluation and datasets. The datasets offered a healthy variety for typical image recognition tasks. It would be interesting to see what these metrics would reveal when applied to other types of data (e.g. scientific images).
The  authors explain, with graphs, the results of the loss, normalization, and architectures. I feel the discussion on loss was rushed, and I gained no insight on what the authors thought was a prominent difference between the three losses studied. Perhaps the authors had no salient observations for loss, but explicitly stating such would be useful to the reader. The only observation I gained as far as this is that non-saturating loss would possibly be stable across various datasets.
Regularization and normalization are discussed in much more detail, and I think the authors made helpful and interesting observations, such as the benefits of spectral normalization and the fact that batch normalization in the discriminator might be a harmful thing. These are good takeaways that could be useful to a vast number of GANs researchers.
For architectures to be a main pillar of the paper, I feel that this area could have been explored in greater detail. I feel that this discussion devolved into a discussion, again, about normalization rather than the architectural differences in performance. Unless I am misunderstanding something, it seems that the authors simply tested one more architecture, for the express purpose of testing whether their observations about normalization would hold.
As a bonus, the authors bring up some problems they had in making comparisons and reproducing results. I think this is an extremely important discussion to have, and I am glad that the authors detailed the obstacles in their journey. Hopefully this will inspire other researchers to avoid adding to the complications in this field.
The graphs were difficult to parse. I was able to make them out, but perhaps separating the top row (FID and diversity graphs) into separate figures, separate lines, or something would have reduced some confusion. In addition, different charts presenting only one loss function, with their spectral normalization and gradient penalty variants, would have made the effects of the normalization more obvious on the FID distribution graphs. If this can be changed before publication, I would strongly suggest it.
I appreciate that the authors provided source code via GitHub. However, in the future, the authors should be careful to provide an anonymous repository for review purposes. I had to be careful not to allow myself to focus on the author names which are prominent in the repository readme, and one of whom has his/her name in the GitHub URL itself. I didn't immediately recognize the names and thus it was easy for me not to retain them or focus on them. However, if it had been otherwise, it might have risked biasing the review.
In all, I think this is a good and useful paper from which I have learned and to which I will refer in the future as I continue my research into GANs and VAEs. I would suggest changing the title to be more appropriate and accurate (the researchers are primarily focused on showing the positive and negative effects of normalization across various loss functions and architectures). But altogether, I believe this is a paper worth publishing at ICLR.