The GAN Landscape: Losses, Architectures, Regularization, and Normalization
Generative adversarial networks (GANs) are a class of deep generative models which aim to learn a target distribution in an unsupervised fashion. While they were successfully applied to many problems, training a GAN is a notoriously challenging task and requires a significant amount of hyperparameter tuning, neural architecture engineering, and a non-trivial amount of ``tricks". The success in many practical applications coupled with the lack of a measure to quantify the failure modes of GANs resulted in a plethora of proposed losses, regularization and normalization schemes, and neural architectures. In this work we take a sober view of the current state of GANs from a practical perspective. We reproduce the current state of the art and go beyond fairly exploring the GAN landscape. We discuss common pitfalls and reproducibility issues, open-source our code on Github, and provide pre-trained models on TensorFlow Hub.
 (As a disclamer I want to point out I*m not an expert in GANs and have only a basic understanding of the sub-field, but arguably this would make me target audience of this paper). The authors presents a large scale study comparing a large number of GAN experiments, in this study they compare various choices of architechtures, losses and hyperparameters. The first part of the paper describes the various losses, architectures, regularization and normalization schemes; and the second part describes the results of the comparison experiments. While I wish there were more such studies -- as I believe reproducing past results experimentally is important, and so is providing practical advice for practitioners -- this work in many parts hard to follow, and it is hard to get lot of new insight from the results, or a better understanding of GANs. As far I can see the most important take home message of the paper can be summarized in "one should consider non-saturating GAN loss and spectral normalization as default choices [...] Given additional computational budget, we suggest adding the gradient penalty [...] and train the model until convergence". Pros: - available source code - large number of experiments Cons: - the exposition could be improved, in particular the description of the plots is not very clear, I*m still not sure exactly what they show - not clear what the target audience of the first part (section 2) is, it is too technical for a survey intended for outsiders, and discusses subtle points that are not easy to understand without more knowledge, but at the same time seems unlikely to give additional insight to an insider - limited amount of new insight, which is limiting as new and better understanding of GANs and practical guidelines are arguably the main contribution of a work of this type Some suggestions that I think could make the paper stronger - I believe that in particular section 2 goes into too many mathematical details and subtleties that do not really add a lot. I think that either the reader already understand those concepts well (which I admit, I don*t really, I*m merely curious about GANs and have been following the action from a distance, hence my low confidence rating to this review), or if they does not, it will be very hard to get much out of it. I would leave out some of the details, shortening the whole sections, and focus more on making a few of the concepts more understandable, and potentially leaving more space for a clearer description of the results - it is not really clear to be what data the graphs show: the boxplots show 5% of what data? does it also include the models obtained by gaussian process regression? and what about the line plots, is it the best model so far as you train more and more models? if so, how are those models chosen and ordered? are they the results of single models or average of multiple ones? - "the variance of models obtained by Guassian Process regression is handled implicitely so we tran each model once"? I do not understand what this means, and I work with hyper-parameter tuning using gaussian processes daily. It should probably be rephrased - at the start of section 3: what is an "experiment"? - in 3.1 towards the end of the first paragraph, what is a "study", is that the same as experiment or something different? - (minor) stating that lower is better in the graphs might be useful - (minor) typo in page 5 "We use a fixed the number"
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
The GAN Landscape: Losses, Architectures, Regularization, and Normalization
Generative adversarial networks (GANs) are a class of deep generative models which aim to learn a target distribution in an unsupervised fashion. While they were successfully applied to many problems, training a GAN is a notoriously challenging task and requires a significant amount of hyperparameter tuning, neural architecture engineering, and a non-trivial amount of ``tricks". The success in many practical applications coupled with the lack of a measure to quantify the failure modes of GANs resulted in a plethora of proposed losses, regularization and normalization schemes, and neural architectures. In this work we take a sober view of the current state of GANs from a practical perspective. We reproduce the current state of the art and go beyond fairly exploring the GAN landscape. We discuss common pitfalls and reproducibility issues, open-source our code on Github, and provide pre-trained models on TensorFlow Hub.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 (As a disclamer I want to point out I*m not an expert in GANs and have only a basic understanding of the sub-field, but arguably this would make me target audience of this paper). The authors presents a large scale study comparing a large number of GAN experiments, in this study they compare various choices of architechtures, losses and hyperparameters. The first part of the paper describes the various losses, architectures, regularization and normalization schemes; and the second part describes the results of the comparison experiments. While I wish there were more such studies -- as I believe reproducing past results experimentally is important, and so is providing practical advice for practitioners -- this work in many parts hard to follow, and it is hard to get lot of new insight from the results, or a better understanding of GANs. As far I can see the most important take home message of the paper can be summarized in "one should consider non-saturating GAN loss and spectral normalization as default choices [...] Given additional computational budget, we suggest adding the gradient penalty [...] and train the model until convergence". Pros: - available source code - large number of experiments Cons: - the exposition could be improved, in particular the description of the plots is not very clear, I*m still not sure exactly what they show - not clear what the target audience of the first part (section 2) is, it is too technical for a survey intended for outsiders, and discusses subtle points that are not easy to understand without more knowledge, but at the same time seems unlikely to give additional insight to an insider - limited amount of new insight, which is limiting as new and better understanding of GANs and practical guidelines are arguably the main contribution of a work of this type Some suggestions that I think could make the paper stronger - I believe that in particular section 2 goes into too many mathematical details and subtleties that do not really add a lot. I think that either the reader already understand those concepts well (which I admit, I don*t really, I*m merely curious about GANs and have been following the action from a distance, hence my low confidence rating to this review), or if they does not, it will be very hard to get much out of it. I would leave out some of the details, shortening the whole sections, and focus more on making a few of the concepts more understandable, and potentially leaving more space for a clearer description of the results - it is not really clear to be what data the graphs show: the boxplots show 5% of what data? does it also include the models obtained by gaussian process regression? and what about the line plots, is it the best model so far as you train more and more models? if so, how are those models chosen and ordered? are they the results of single models or average of multiple ones? - "the variance of models obtained by Guassian Process regression is handled implicitely so we tran each model once"? I do not understand what this means, and I work with hyper-parameter tuning using gaussian processes daily. It should probably be rephrased - at the start of section 3: what is an "experiment"? - in 3.1 towards the end of the first paragraph, what is a "study", is that the same as experiment or something different? - (minor) stating that lower is better in the graphs might be useful - (minor) typo in page 5 "We use a fixed the number"
assistant
 (As a disclamer I want to point out I*m not an expert in GANs and have only a basic understanding of the sub-field, but arguably this would make me target audience of this paper). The authors presents a large scale study comparing a large number of GAN experiments, in this study they compare various choices of architechtures, losses and hyperparameters. The first part of the paper describes the various losses, architectures, regularization and normalization schemes; and the second part describes the results of the comparison experiments. While I wish there were more such studies -- as I believe reproducing past results experimentally is important, and so is providing practical advice for practitioners -- this work in many parts hard to follow, and it is hard to get lot of new insight from the results, or a better understanding of GANs. As far I can see the most important take home message of the paper can be summarized in "one should consider non-saturating GAN loss and spectral normalization as default choices [...] Given additional computational budget, we suggest adding the gradient penalty [...] and train the model until convergence". Pros: - available source code - large number of experiments Cons: - the exposition could be improved, in particular the description of the plots is not very clear, I*m still not sure exactly what they show - not clear what the target audience of the first part (section 2) is, it is too technical for a survey intended for outsiders, and discusses subtle points that are not easy to understand without more knowledge, but at the same time seems unlikely to give additional insight to an insider - limited amount of new insight, which is limiting as new and better understanding of GANs and practical guidelines are arguably the main contribution of a work of this type Some suggestions that I think could make the paper stronger - I believe that in particular section 2 goes into too many mathematical details and subtleties that do not really add a lot. I think that either the reader already understand those concepts well (which I admit, I don*t really, I*m merely curious about GANs and have been following the action from a distance, hence my low confidence rating to this review), or if they does not, it will be very hard to get much out of it. I would leave out some of the details, shortening the whole sections, and focus more on making a few of the concepts more understandable, and potentially leaving more space for a clearer description of the results - it is not really clear to be what data the graphs show: the boxplots show 5% of what data? does it also include the models obtained by gaussian process regression? and what about the line plots, is it the best model so far as you train more and more models? if so, how are those models chosen and ordered? are they the results of single models or average of multiple ones? - "the variance of models obtained by Guassian Process regression is handled implicitely so we tran each model once"? I do not understand what this means, and I work with hyper-parameter tuning using gaussian processes daily. It should probably be rephrased - at the start of section 3: what is an "experiment"? - in 3.1 towards the end of the first paragraph, what is a "study", is that the same as experiment or something different? - (minor) stating that lower is better in the graphs might be useful - (minor) typo in page 5 "We use a fixed the number"