An empirical study on evaluation metrics of generative adversarial networks
Despite the widespread interest in generative adversarial networks (GANs), few works have studied the metrics that quantitatively evaluate GANs* performance. In this paper, we revisit several representative sample-based evaluation metrics for GANs, and address the important problem of emph{how to evaluate the evaluation metrics}. We start with a few necessary conditions for metrics to produce meaningful scores, such as distinguishing real from generated samples, identifying mode dropping and mode collapsing, and detecting overfitting. Then with a series of carefully designed experiments, we are able to comprehensively investigate existing sample-based metrics and identify their strengths and limitations in practical settings. Based on these results, we observe that kernel Maximum Mean Discrepancy (MMD) and the 1-Nearest-Neighbour (1-NN) two-sample test seem to satisfy most of the desirable properties, provided that the distances between samples are computed in a suitable feature space. Our experiments also unveil interesting properties about the behavior of several popular GAN models, such as whether they are memorizing training samples, and how far these state-of-the-art GANs are from perfect.
 In the paper, the authors discuss several GAN evaluation metrics. Specifically, the authors pointed out some desirable properties that GANS evaluation metrics should satisfy. For those properties raised, the authors experimentally evaluated whether existing metrics satisfy those properties or not. Section 4 summarizes the results, which concluded that the Kernel MMD and 1-NN classifier in the feature space are so far recommended metrics to be used. I think this paper tackles an interesting and important problem, what metrics are preferred for evaluating GANs. In particular, the authors showed that Inception Score, which is one of the most popular metric, is actually not preferred for several reasons. The result, comparing data distributions and the distribution of the generator would be the preferred choice (that can be attained by Kernel MMD and 1-NN classifier), seems to be reasonable. This would not be a surprising result as the ultimate goal of GAN is mimicking the data distribution. However, the result is supported by exhaustive experiments making the result highly convincing. Overall, I think this paper is worthy for acceptance as several GAN methods are proposed and good evaluation metrics are needed for further improvements of the research field.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
An empirical study on evaluation metrics of generative adversarial networks
Despite the widespread interest in generative adversarial networks (GANs), few works have studied the metrics that quantitatively evaluate GANs* performance. In this paper, we revisit several representative sample-based evaluation metrics for GANs, and address the important problem of emph{how to evaluate the evaluation metrics}. We start with a few necessary conditions for metrics to produce meaningful scores, such as distinguishing real from generated samples, identifying mode dropping and mode collapsing, and detecting overfitting. Then with a series of carefully designed experiments, we are able to comprehensively investigate existing sample-based metrics and identify their strengths and limitations in practical settings. Based on these results, we observe that kernel Maximum Mean Discrepancy (MMD) and the 1-Nearest-Neighbour (1-NN) two-sample test seem to satisfy most of the desirable properties, provided that the distances between samples are computed in a suitable feature space. Our experiments also unveil interesting properties about the behavior of several popular GAN models, such as whether they are memorizing training samples, and how far these state-of-the-art GANs are from perfect.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 In the paper, the authors discuss several GAN evaluation metrics. Specifically, the authors pointed out some desirable properties that GANS evaluation metrics should satisfy. For those properties raised, the authors experimentally evaluated whether existing metrics satisfy those properties or not. Section 4 summarizes the results, which concluded that the Kernel MMD and 1-NN classifier in the feature space are so far recommended metrics to be used. I think this paper tackles an interesting and important problem, what metrics are preferred for evaluating GANs. In particular, the authors showed that Inception Score, which is one of the most popular metric, is actually not preferred for several reasons. The result, comparing data distributions and the distribution of the generator would be the preferred choice (that can be attained by Kernel MMD and 1-NN classifier), seems to be reasonable. This would not be a surprising result as the ultimate goal of GAN is mimicking the data distribution. However, the result is supported by exhaustive experiments making the result highly convincing. Overall, I think this paper is worthy for acceptance as several GAN methods are proposed and good evaluation metrics are needed for further improvements of the research field.
assistant
 In the paper, the authors discuss several GAN evaluation metrics. Specifically, the authors pointed out some desirable properties that GANS evaluation metrics should satisfy. For those properties raised, the authors experimentally evaluated whether existing metrics satisfy those properties or not. Section 4 summarizes the results, which concluded that the Kernel MMD and 1-NN classifier in the feature space are so far recommended metrics to be used. I think this paper tackles an interesting and important problem, what metrics are preferred for evaluating GANs. In particular, the authors showed that Inception Score, which is one of the most popular metric, is actually not preferred for several reasons. The result, comparing data distributions and the distribution of the generator would be the preferred choice (that can be attained by Kernel MMD and 1-NN classifier), seems to be reasonable. This would not be a surprising result as the ultimate goal of GAN is mimicking the data distribution. However, the result is supported by exhaustive experiments making the result highly convincing. Overall, I think this paper is worthy for acceptance as several GAN methods are proposed and good evaluation metrics are needed for further improvements of the research field.