An empirical study on evaluation metrics of generative adversarial networks
Despite the widespread interest in generative adversarial networks (GANs), few works have studied the metrics that quantitatively evaluate GANs* performance. In this paper, we revisit several representative sample-based evaluation metrics for GANs, and address the important problem of emph{how to evaluate the evaluation metrics}. We start with a few necessary conditions for metrics to produce meaningful scores, such as distinguishing real from generated samples, identifying mode dropping and mode collapsing, and detecting overfitting. Then with a series of carefully designed experiments, we are able to comprehensively investigate existing sample-based metrics and identify their strengths and limitations in practical settings. Based on these results, we observe that kernel Maximum Mean Discrepancy (MMD) and the 1-Nearest-Neighbour (1-NN) two-sample test seem to satisfy most of the desirable properties, provided that the distances between samples are computed in a suitable feature space. Our experiments also unveil interesting properties about the behavior of several popular GAN models, such as whether they are memorizing training samples, and how far these state-of-the-art GANs are from perfect.
 The paper describes an empirical evaluation of some of the most common metrics to evaluate GANs (inception score, mode score, kernel MMD, Wasserstein distance and LOO accuracy). The paper is well written, clear, organized and easy to follow. Given that the underlying application is image generation, the authors move from a pixel representation of images to using the feature representation given by a pre-trained ResNet, which is key in their results and further comparisons. They analyzed discriminability, mode collapsing and dropping, robustness to transformations, efficiency and overfitting. Although this work and its results are very useful for practitioners, it lacks in two aspects. First, it only considers a single task for which GANs are very popular. Second, it could benefit from a deeper (maybe theoretical analysis) of some of the questions. Some of the conclusions could be further clarified with additional experiments (e.g., Sec 3.6 ‘while the reason that RMS also fails to detect overfitting may again be its lack of generalization to datasets with classes not contained in the ImageNet dataset’).
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
An empirical study on evaluation metrics of generative adversarial networks
Despite the widespread interest in generative adversarial networks (GANs), few works have studied the metrics that quantitatively evaluate GANs* performance. In this paper, we revisit several representative sample-based evaluation metrics for GANs, and address the important problem of emph{how to evaluate the evaluation metrics}. We start with a few necessary conditions for metrics to produce meaningful scores, such as distinguishing real from generated samples, identifying mode dropping and mode collapsing, and detecting overfitting. Then with a series of carefully designed experiments, we are able to comprehensively investigate existing sample-based metrics and identify their strengths and limitations in practical settings. Based on these results, we observe that kernel Maximum Mean Discrepancy (MMD) and the 1-Nearest-Neighbour (1-NN) two-sample test seem to satisfy most of the desirable properties, provided that the distances between samples are computed in a suitable feature space. Our experiments also unveil interesting properties about the behavior of several popular GAN models, such as whether they are memorizing training samples, and how far these state-of-the-art GANs are from perfect.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper describes an empirical evaluation of some of the most common metrics to evaluate GANs (inception score, mode score, kernel MMD, Wasserstein distance and LOO accuracy). The paper is well written, clear, organized and easy to follow. Given that the underlying application is image generation, the authors move from a pixel representation of images to using the feature representation given by a pre-trained ResNet, which is key in their results and further comparisons. They analyzed discriminability, mode collapsing and dropping, robustness to transformations, efficiency and overfitting. Although this work and its results are very useful for practitioners, it lacks in two aspects. First, it only considers a single task for which GANs are very popular. Second, it could benefit from a deeper (maybe theoretical analysis) of some of the questions. Some of the conclusions could be further clarified with additional experiments (e.g., Sec 3.6 ‘while the reason that RMS also fails to detect overfitting may again be its lack of generalization to datasets with classes not contained in the ImageNet dataset’).
assistant
 The paper describes an empirical evaluation of some of the most common metrics to evaluate GANs (inception score, mode score, kernel MMD, Wasserstein distance and LOO accuracy). The paper is well written, clear, organized and easy to follow. Given that the underlying application is image generation, the authors move from a pixel representation of images to using the feature representation given by a pre-trained ResNet, which is key in their results and further comparisons. They analyzed discriminability, mode collapsing and dropping, robustness to transformations, efficiency and overfitting. Although this work and its results are very useful for practitioners, it lacks in two aspects. First, it only considers a single task for which GANs are very popular. Second, it could benefit from a deeper (maybe theoretical analysis) of some of the questions. Some of the conclusions could be further clarified with additional experiments (e.g., Sec 3.6 ‘while the reason that RMS also fails to detect overfitting may again be its lack of generalization to datasets with classes not contained in the ImageNet dataset’).