Are GANs Created Equal? A Large-Scale Study
Generative adversarial networks (GAN) are a powerful subclass of generative models. Despite a very rich research activity leading to numerous interesting GAN algorithms, it is still very hard to assess which algorithm(s) perform better than others. We conduct a neutral, multi-faceted large-scale empirical study on state-of-the art models and evaluation measures. We find that most models can reach similar scores with enough hyperparameter optimization and random restarts. This suggests that improvements can arise from a higher computational budget and tuning more than fundamental algorithmic changes. To overcome some limitations of the current metrics, we also propose several data sets on which precision and recall can be computed. Our experimental results suggest that future GAN research should be based on more systematic and objective evaluation procedures. Finally, we did not find evidence that any of the tested algorithms consistently outperforms the non-saturating GAN introduced in [9].
In this paper the authors tackle the non-trivial problem of GANs comparison. Comparing the different approaches toward training GANs is a major issue towards a better understanding of deep generative models.   This paper introduces a methodological procedure for comparing GANS via two different metrics FIA, and Precision, recall and F1. The authors compare 7 Different GANs (different Generative/Discriminator loss, and regularization terms)  under the same network architecture. A procedure for testing different hyperparameters, that allows for different computational budgets, is introduced. The randomness due to network initialization is also analyzed. The experiments are run on four popular datasets, and also a synthetic dataset consisting of binary images of triangles is used. This synthetic dataset allows to directly compute the distance from a generated sample to the considered manifold (via non-convex optimization).  There are two main aspects that show the importance/contributions of the  paper.   Introduces a methodology for comparing GANs (choose a bunch of different loss functions, different regularizations, fix architecture, adopt a metric, choose different datasets, repeat experiments since training involve randomness, introduce the computational budget as a way of quantitatively comparing how difficult is to reach good hyperparameters); Reaches interesting conclusions. As a first example, the paper concludes from empirical evidence that it does not make sense to compare the best FID results. Comparison should involve the distribution of the adopted metric under different runs.  As a second example, the authors show empirically that given the right hyperparameters the best GAN (according to FID) depends on the dataset. There is no clear winner.   I believe this paper will help the community to start a discussion on how to fairly compare deep generative models.  *Major Points*  The authors claim that algorithmic differences in state-of-the-art GANs become less relevant, as the computational budget increases. This is true according to the way the authors carried out the experiments. But, what happens if we start the hyperparameters tuning by doing a random search from the recommended values for each algorithm. Maybe the hyperparameters recommended by the authors are already good enough (or they are close to the best ones) for that particular dataset. For instance, according to the authors of WGAN-GP the same set of hyperparameters worked OK in many different experiments. Please comment.  A clear limitation of the current work (in terms of the conclusions that are drawn) is that only one architecture is considered (infoGAN). All the conclusions are with respect to this architecture. So, one wonders whether this analysis is  general enough. The implicance of this should be commented in the text.  The conclusions are too strong. In fact, the paper shows that the average performance (across datasets) is similar but the best one depends on the dataset. This does not mean that all the GANs will reach the same best performance on every dataset.   Inability to detect overfitting by FID. This is commented in Section 5, but then it is not linked to the rest of the paper. In particular, the finding regarding precision and recall on the synthetic dataset are not commented in the conclusions. I recommend to improve the connection of this experiment to the large-scale experimental evaluation.  *Minor Comments / Typos*  l26  - metric → metrics l40 - Fréchet Inception Distance → Fréchet Inception Distance (FID) l77 - mean and covariance is estimated → mean and covariance are estimated l95 - "... and corresponding loss, and the presence..." → "... and corresponding loss, and (ii) the presence..." l103 - MM GAN  and WGAN should be in \textsc l119 - Add (\textsc{began}) somewhere in the sentence. l93-l120 - What about the different hyperparameters that only a few configurations have: $c$, $k_t$, $\gamma$. l155- In general when refer to a Figure/Table in Appendix please say that it is in the Appendix/Supplementary Material. l225- How bad is to get stuck in a local minima when doing this minimization? It seems that this implies that you are giving an upper bound on the performance (i.e., false negative rate might be lower). How many times did you repeat from different initializations. Please comment. l230- Which dimension did you consider for the latent space (d_z)? l239 - " as a popular alternative" → "as a popular alternative to GANs" l294- "with distance below" → "with distance to the manifold below"   **Updated After Rebuttal**  I see three main limitations of the paper: 1. Only one architecture is tested, 2. Hyperparameters search is not completely fair (in rebuttal the authors mention sequential Bayesian optimization as FW on this matter), 3. Very hard to replicate due to the large amount of computational resources needed.  Even with 1. and 2., I think the paper worths publication since one can build on this work to compare different architectures and more rational ways of searching for hyperparameters. Regarding 3, the authors open sourced code and trained models which is a step forward on reproducibility. Finally the current conclusions of the paper are too strong (this point needs to be revised).