Towards a Testable Notion of Generalization for Generative Adversarial Networks
We consider the question of how to assess generative adversarial networks, in particular with respect to whether or not they generalise beyond memorising the training data. We propose a simple procedure for assessing generative adversarial network performance based on a principled consideration of what the actual goal of generalisation is. Our approach involves using a test set to estimate the Wasserstein distance between the generative distribution produced by our procedure, and the underlying data distribution. We use this procedure to assess the performance of several modern generative adversarial network architectures. We find that this procedure is sensitive to the choice of ground metric on the underlying data space, and suggest a choice of ground metric that substantially improves performance. We finally suggest that attending to the ground metric used in Wasserstein generative adversarial network training may be fruitful, and outline a concrete pathway towards doing so.
 The quality of the paper is good, and clarity is mostly good. The proposed metric is interesting, but it is hard to judge the significance without more thorough experiments demonstrating that it works in practice. Pros: - clear definitions of terms - overall outline of paper is good - novel metric Cons - text is a bit over-wordy, and flow/meaning sometimes get lost. A strict editor would be helpful, because the underlying content is good - odd that your definition of generalization in GANs appears immediately preceding the section titled "Generalisation in GANs" - the paragraph at the end of the "Generalisation in GANs" section is confusing. I think this section and the previous ("The objective of unsupervised learning") could be combined, removing some repetition, adding some subtitles to improve clarity. This would cut down the text a bit to make space for more experiments. - why is your definition of generalization that the test set distance is strictly less than training set ? I would think this should be less-than-or-equal - there is a sentence that doesn*t end at the top of p.3: "... the original GAN paper showed that [ends here]" - should state in the abstract what your "notion of generalization" for gans is, instead of being vague about it - more experiments showing a comparison of the proposed metric to others (e.g. inception score, Mturk assessments of sample quality, etc.) would be necessary to find the metric convincing - what is a "pushforward measure"? (p.2) - the related work section is well-written and interesting, but it*s a bit odd to have it at the end. Earlier in the work (e.g. before experiments and discussion) would allow the comparison with MMD to inform the context of the introduction - there are some errors in figures that I think were all mentioned by previous commentators.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Towards a Testable Notion of Generalization for Generative Adversarial Networks
We consider the question of how to assess generative adversarial networks, in particular with respect to whether or not they generalise beyond memorising the training data. We propose a simple procedure for assessing generative adversarial network performance based on a principled consideration of what the actual goal of generalisation is. Our approach involves using a test set to estimate the Wasserstein distance between the generative distribution produced by our procedure, and the underlying data distribution. We use this procedure to assess the performance of several modern generative adversarial network architectures. We find that this procedure is sensitive to the choice of ground metric on the underlying data space, and suggest a choice of ground metric that substantially improves performance. We finally suggest that attending to the ground metric used in Wasserstein generative adversarial network training may be fruitful, and outline a concrete pathway towards doing so.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The quality of the paper is good, and clarity is mostly good. The proposed metric is interesting, but it is hard to judge the significance without more thorough experiments demonstrating that it works in practice. Pros: - clear definitions of terms - overall outline of paper is good - novel metric Cons - text is a bit over-wordy, and flow/meaning sometimes get lost. A strict editor would be helpful, because the underlying content is good - odd that your definition of generalization in GANs appears immediately preceding the section titled "Generalisation in GANs" - the paragraph at the end of the "Generalisation in GANs" section is confusing. I think this section and the previous ("The objective of unsupervised learning") could be combined, removing some repetition, adding some subtitles to improve clarity. This would cut down the text a bit to make space for more experiments. - why is your definition of generalization that the test set distance is strictly less than training set ? I would think this should be less-than-or-equal - there is a sentence that doesn*t end at the top of p.3: "... the original GAN paper showed that [ends here]" - should state in the abstract what your "notion of generalization" for gans is, instead of being vague about it - more experiments showing a comparison of the proposed metric to others (e.g. inception score, Mturk assessments of sample quality, etc.) would be necessary to find the metric convincing - what is a "pushforward measure"? (p.2) - the related work section is well-written and interesting, but it*s a bit odd to have it at the end. Earlier in the work (e.g. before experiments and discussion) would allow the comparison with MMD to inform the context of the introduction - there are some errors in figures that I think were all mentioned by previous commentators.
assistant
 The quality of the paper is good, and clarity is mostly good. The proposed metric is interesting, but it is hard to judge the significance without more thorough experiments demonstrating that it works in practice. Pros: - clear definitions of terms - overall outline of paper is good - novel metric Cons - text is a bit over-wordy, and flow/meaning sometimes get lost. A strict editor would be helpful, because the underlying content is good - odd that your definition of generalization in GANs appears immediately preceding the section titled "Generalisation in GANs" - the paragraph at the end of the "Generalisation in GANs" section is confusing. I think this section and the previous ("The objective of unsupervised learning") could be combined, removing some repetition, adding some subtitles to improve clarity. This would cut down the text a bit to make space for more experiments. - why is your definition of generalization that the test set distance is strictly less than training set ? I would think this should be less-than-or-equal - there is a sentence that doesn*t end at the top of p.3: "... the original GAN paper showed that [ends here]" - should state in the abstract what your "notion of generalization" for gans is, instead of being vague about it - more experiments showing a comparison of the proposed metric to others (e.g. inception score, Mturk assessments of sample quality, etc.) would be necessary to find the metric convincing - what is a "pushforward measure"? (p.2) - the related work section is well-written and interesting, but it*s a bit odd to have it at the end. Earlier in the work (e.g. before experiments and discussion) would allow the comparison with MMD to inform the context of the introduction - there are some errors in figures that I think were all mentioned by previous commentators.