Revisiting Classifier Two-Sample Tests
The goal of two-sample tests is to assess whether two samples, and , are drawn from the same distribution. Perhaps intriguingly, one relatively unexplored method to build two-sample tests is the use of binary classifiers. In particular, construct a dataset by pairing the examples in with a positive label, and by pairing the examples in with a negative label. If the null hypothesis `` ** is true, then the classification accuracy of a binary classifier on a held-out subset of this dataset should remain near chance-level. As we will show, such emph{Classifier Two-Sample Tests} (C2ST) learn a suitable representation of the data on the fly, return test statistics in interpretable units, have a simple null distribution, and their predictive uncertainty allow to interpret where and differ. The goal of this paper is to establish the properties, performance, and uses of C2ST. First, we analyze their main theoretical properties. Second, we compare their performance against a variety of state-of-the-art alternatives. Third, we propose their use to evaluate the sample quality of generative models with intractable likelihoods, such as Generative Adversarial Networks (GANs). Fourth, we showcase the novel application of GANs together with C2ST for causal discovery.
The submission considers the setting of 2-sample testing from the perspective of evaluating a classifier. For a classifier between two samples from the same distribution, the distribution of the classification accuracy follows a simple form under the null hypothesis. As such, a straightforward threshold can be derived for any classifier. Finding a more powerful test then amounts to training a better classifier. One may then focus efforts, e.g. on deep neural networks, for which statistics such as the MMD may be very difficult to characterize. + The approach is sound and very general + The paper is timely in that deep learning has had huge impacts in classification and other prediction settings, but has not had as big an impact on statistical hypothesis testing as kernel methods have - The discussion of the relationship to kernel-MMD has not always been as realistic as it could have been. For example, the kernel-MMD can also be seen as a classifier based approach, so a more fair discussion could be provided. Also, the form of kernel-MMD used in the comparisons is a bit contradictory to the discussion as well * The linear kernel-MMD is used which is less powerful than the quadradic kernel-MMD (the authors have justified this from the perspective of computation time) * The kernel-MMD is argued against due to its unwieldy distribution under the null, but the linear time kernel-MMD (see also Zaremba et al., NIPS 2013) has a Gaussian distribution under the null. Arthur Gretton*s comment from Dec 14 during the discussion period was very insightful and helpful. If these insights and additional experiments comparing the kernel-MMD to the classifier threshold on the blobs dataset could be included, that would be very helpful for understanding the paper. The open review format gives an excellent opportunity to assign proper credit for these experiments and insights by citing the comment.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Revisiting Classifier Two-Sample Tests
The goal of two-sample tests is to assess whether two samples, and , are drawn from the same distribution. Perhaps intriguingly, one relatively unexplored method to build two-sample tests is the use of binary classifiers. In particular, construct a dataset by pairing the examples in with a positive label, and by pairing the examples in with a negative label. If the null hypothesis `` ** is true, then the classification accuracy of a binary classifier on a held-out subset of this dataset should remain near chance-level. As we will show, such emph{Classifier Two-Sample Tests} (C2ST) learn a suitable representation of the data on the fly, return test statistics in interpretable units, have a simple null distribution, and their predictive uncertainty allow to interpret where and differ. The goal of this paper is to establish the properties, performance, and uses of C2ST. First, we analyze their main theoretical properties. Second, we compare their performance against a variety of state-of-the-art alternatives. Third, we propose their use to evaluate the sample quality of generative models with intractable likelihoods, such as Generative Adversarial Networks (GANs). Fourth, we showcase the novel application of GANs together with C2ST for causal discovery.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The submission considers the setting of 2-sample testing from the perspective of evaluating a classifier. For a classifier between two samples from the same distribution, the distribution of the classification accuracy follows a simple form under the null hypothesis. As such, a straightforward threshold can be derived for any classifier. Finding a more powerful test then amounts to training a better classifier. One may then focus efforts, e.g. on deep neural networks, for which statistics such as the MMD may be very difficult to characterize. + The approach is sound and very general + The paper is timely in that deep learning has had huge impacts in classification and other prediction settings, but has not had as big an impact on statistical hypothesis testing as kernel methods have - The discussion of the relationship to kernel-MMD has not always been as realistic as it could have been. For example, the kernel-MMD can also be seen as a classifier based approach, so a more fair discussion could be provided. Also, the form of kernel-MMD used in the comparisons is a bit contradictory to the discussion as well * The linear kernel-MMD is used which is less powerful than the quadradic kernel-MMD (the authors have justified this from the perspective of computation time) * The kernel-MMD is argued against due to its unwieldy distribution under the null, but the linear time kernel-MMD (see also Zaremba et al., NIPS 2013) has a Gaussian distribution under the null. Arthur Gretton*s comment from Dec 14 during the discussion period was very insightful and helpful. If these insights and additional experiments comparing the kernel-MMD to the classifier threshold on the blobs dataset could be included, that would be very helpful for understanding the paper. The open review format gives an excellent opportunity to assign proper credit for these experiments and insights by citing the comment.
assistant
The submission considers the setting of 2-sample testing from the perspective of evaluating a classifier. For a classifier between two samples from the same distribution, the distribution of the classification accuracy follows a simple form under the null hypothesis. As such, a straightforward threshold can be derived for any classifier. Finding a more powerful test then amounts to training a better classifier. One may then focus efforts, e.g. on deep neural networks, for which statistics such as the MMD may be very difficult to characterize. + The approach is sound and very general + The paper is timely in that deep learning has had huge impacts in classification and other prediction settings, but has not had as big an impact on statistical hypothesis testing as kernel methods have - The discussion of the relationship to kernel-MMD has not always been as realistic as it could have been. For example, the kernel-MMD can also be seen as a classifier based approach, so a more fair discussion could be provided. Also, the form of kernel-MMD used in the comparisons is a bit contradictory to the discussion as well * The linear kernel-MMD is used which is less powerful than the quadradic kernel-MMD (the authors have justified this from the perspective of computation time) * The kernel-MMD is argued against due to its unwieldy distribution under the null, but the linear time kernel-MMD (see also Zaremba et al., NIPS 2013) has a Gaussian distribution under the null. Arthur Gretton*s comment from Dec 14 during the discussion period was very insightful and helpful. If these insights and additional experiments comparing the kernel-MMD to the classifier threshold on the blobs dataset could be included, that would be very helpful for understanding the paper. The open review format gives an excellent opportunity to assign proper credit for these experiments and insights by citing the comment.