Towards GAN Benchmarks Which Require Generalization
For many evaluation metrics commonly used as benchmarks for unconditional image generation, trivially memorizing the training set attains a better score than models which are considered state-of-the-art; we consider this problematic. We clarify a necessary condition for an evaluation metric not to behave this way: estimating the function must require a large sample from the model. In search of such a metric, we turn to neural network divergences (NNDs), which are defined in terms of a neural network trained to distinguish between distributions. The resulting benchmarks cannot be ``won** by training set memorization, while still being perceptually correlated and computable only from samples. We survey past work on using NNDs for evaluation, implement an example black-box metric based on these ideas, and validate experimentally that it can measure a notion of generalization.
 This paper is quite interesting as it tries to find a new metric for evaluating GANs. IS is a terrible metric, as memorization would achieve high score and test log-likelihood cannot be evaluated. I like the long discussion at the beginning of the paper about what a metric for evaluating implicit generative models would need to be a valid and useful metric. This problem is of great importance for GANs as proving that GANs solve the density estimation problem would be extremely hard and even more so, making sure we are close to a good solution with any finite sample even more so (I am talking to non-trivial examples in high dimensions). It is clear that in order to make GANs, in particular, or implicit models, in general, useful, we need to find metrics that would allow us to achieve progress. This paper is a direction in what it needed. In this sense I think the paper can be a good starting point for the discussion that we are not having right now, because we are too focused on making sure they converge, but not how they can be useful. On the down side, I think the proposed DNN metric is not exactly useful. It would be a subset of the metric that an MMD would give and it would focus only in some properties of the images but not on the whole distribution. So, if this metric does not capture the relevant aspects of the problem the GAN is trying to imitate, it will fail to provide that metric that we are looking for. I would see this paper as a great workshop paper, in the sense of old-fashion NIPS workshops in which new ideas were tested and discussed. But it clearly would like the polished papers that we see in conferences these days. Bernhard Schoelkopf told me once, after receiving the ICML reviews, “People now focus more on reasons to reject a paper than in reason for accepting a paper.” (note that I am quoting from memory, the bad use of English in mine not his). There are many reasons to reject this paper, but also some reason to accept the paper.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Towards GAN Benchmarks Which Require Generalization
For many evaluation metrics commonly used as benchmarks for unconditional image generation, trivially memorizing the training set attains a better score than models which are considered state-of-the-art; we consider this problematic. We clarify a necessary condition for an evaluation metric not to behave this way: estimating the function must require a large sample from the model. In search of such a metric, we turn to neural network divergences (NNDs), which are defined in terms of a neural network trained to distinguish between distributions. The resulting benchmarks cannot be ``won** by training set memorization, while still being perceptually correlated and computable only from samples. We survey past work on using NNDs for evaluation, implement an example black-box metric based on these ideas, and validate experimentally that it can measure a notion of generalization.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper is quite interesting as it tries to find a new metric for evaluating GANs. IS is a terrible metric, as memorization would achieve high score and test log-likelihood cannot be evaluated. I like the long discussion at the beginning of the paper about what a metric for evaluating implicit generative models would need to be a valid and useful metric. This problem is of great importance for GANs as proving that GANs solve the density estimation problem would be extremely hard and even more so, making sure we are close to a good solution with any finite sample even more so (I am talking to non-trivial examples in high dimensions). It is clear that in order to make GANs, in particular, or implicit models, in general, useful, we need to find metrics that would allow us to achieve progress. This paper is a direction in what it needed. In this sense I think the paper can be a good starting point for the discussion that we are not having right now, because we are too focused on making sure they converge, but not how they can be useful. On the down side, I think the proposed DNN metric is not exactly useful. It would be a subset of the metric that an MMD would give and it would focus only in some properties of the images but not on the whole distribution. So, if this metric does not capture the relevant aspects of the problem the GAN is trying to imitate, it will fail to provide that metric that we are looking for. I would see this paper as a great workshop paper, in the sense of old-fashion NIPS workshops in which new ideas were tested and discussed. But it clearly would like the polished papers that we see in conferences these days. Bernhard Schoelkopf told me once, after receiving the ICML reviews, “People now focus more on reasons to reject a paper than in reason for accepting a paper.” (note that I am quoting from memory, the bad use of English in mine not his). There are many reasons to reject this paper, but also some reason to accept the paper.
assistant
 This paper is quite interesting as it tries to find a new metric for evaluating GANs. IS is a terrible metric, as memorization would achieve high score and test log-likelihood cannot be evaluated. I like the long discussion at the beginning of the paper about what a metric for evaluating implicit generative models would need to be a valid and useful metric. This problem is of great importance for GANs as proving that GANs solve the density estimation problem would be extremely hard and even more so, making sure we are close to a good solution with any finite sample even more so (I am talking to non-trivial examples in high dimensions). It is clear that in order to make GANs, in particular, or implicit models, in general, useful, we need to find metrics that would allow us to achieve progress. This paper is a direction in what it needed. In this sense I think the paper can be a good starting point for the discussion that we are not having right now, because we are too focused on making sure they converge, but not how they can be useful. On the down side, I think the proposed DNN metric is not exactly useful. It would be a subset of the metric that an MMD would give and it would focus only in some properties of the images but not on the whole distribution. So, if this metric does not capture the relevant aspects of the problem the GAN is trying to imitate, it will fail to provide that metric that we are looking for. I would see this paper as a great workshop paper, in the sense of old-fashion NIPS workshops in which new ideas were tested and discussed. But it clearly would like the polished papers that we see in conferences these days. Bernhard Schoelkopf told me once, after receiving the ICML reviews, “People now focus more on reasons to reject a paper than in reason for accepting a paper.” (note that I am quoting from memory, the bad use of English in mine not his). There are many reasons to reject this paper, but also some reason to accept the paper.