We show how to adjust for the variance introduced by dropout with corrections to weight initialization and Batch Normalization, yielding higher accuracy. Though dropout can preserve the expected input to a neuron between train and test, the variance of the input differs. We thus propose a new weight initialization by correcting for the influence of dropout rates and an arbitrary nonlinearity’s influence on variance through simple corrective scalars. Since Batch Normalization trained with dropout estimates the variance of a layer’s incoming distribution with some inputs dropped, the variance also differs between train and test. After training a network with Batch Normalization and dropout, we simply update Batch Normalization’s variance moving averages with dropout off and obtain state of the art on CIFAR-10 and CIFAR-100 without data augmentation.
We show how to adjust for the variance introduced by dropout with corrections to weight initialization and Batch Normalization, yielding higher accuracy. Though dropout can preserve the expected input to a neuron between train and test, the variance of the input differs. We thus propose a new weight initialization by correcting for the influence of dropout rates and an arbitrary nonlinearity’s influence on variance through simple corrective scalars. Since Batch Normalization trained with dropout estimates the variance of a layer’s incoming distribution with some inputs dropped, the variance also differs between train and test. After training a network with Batch Normalization and dropout, we simply update Batch Normalization’s variance moving averages with dropout off and obtain state of the art on CIFAR-10 and CIFAR-100 without data augmentation.
1 INTRODUCTION
Weight initialization and Batch Normalization greatly influence a neural network’s ability to learn. Both methods can allow for a unit-variance neuron input distribution. This is desirable because variance larger or smaller than one may cause activation outputs to explode or vanish. In order to encourage unit-variance, early weight initialization attempts sought to adjust for a neuron’s fan-in (LeCun et al., 1998). More recent initializations correct for a neuron’s fan-out (Glorot & Bengio, 2010). Meanwhile, some weight initializations compensate for the compressiveness of the ReLU nonlinearity (the ReLU’s tendency to reduce output variance) (He et al., 2015). Indeed, He et al. (2015) also show that initializations without a specific, small corrective factor can render a neural network untrainable. To address this issue Batch Normalization (Ioffe & Szegedy, 2015) reduces the role of weight initialization at the cost of up to 30% more computation (Mishkin & Matas, 2015). A less computationally expensive solution is the LSUV weight initialization, yet this still requires computing batch statistics, a special forward pass, and makes no adjustment for backpropagation error signal variance (Mishkin & Matas, 2015). Similarly, weight normalization uses a special feedforward pass and computes batch statistics (Salimans & Kingma, 2016). The continued development of variance stabilizing techniques testifies to its importance for neural networks.
Both Batch Normalization and previous weight initializations do not accommodate the variance introduced by dropout, and we contribute methods to fix this. First we demonstrate a new weight initialization technique which includes a new correction factor for a layer’s dropout rate and adjusts for an arbitrary nonlinearity’s effect on the neuron output variance. All of this is obtained without computing batch statistics or special adjustments to the forward pass, unlike recent methods to control variance (Ioffe & Szegedy, 2015; Mishkin & Matas, 2015; Salimans & Kingma, 2016). By this new initialization, we enable faster and more accurate convergence. Afterward, we show that networks trained with Batch Normalization can improve their accuracy by adjusting for dropout’s variance. We accomplish this by training a network with both Batch Normalization and dropout, then after training we feed forward the training dataset with dropout off to reestimate the Batch Normalization variance estimates. Because of this simple, general technique, we obtain state of the art on CIFAR-10 and CIFAR-100 without data augmentation.
∗Work done while the author was at TTIC. Code available at github.com/hendrycks/init
2 WEIGHT INITIALIZATION
2.1 DERIVATION
In this section, we derive our new initialization by considering a neuron input distribution and its major sources of variance. We accomplish this by separately considering the feedforward and the backpropagation stages.
2.1.1 THE FORWARD PASS
We use f to denote the pointwise nonlinearity in each neural network layer. For simplicity, we use the term “neuron” to refer to an entry in a layer before applying f . Let us also call the input of the l-th layer zl−1, and let the nin × nout weight matrix W l map from layer l − 1 to l. Let an entry of this matrix be wl. In our upcoming initialization, we initialize each column of W l on the unit hypersphere so that each column has an `2 norm of 1. Now, if we assume that this network is trained with a dropout keep rate of p, we must scale the output of a layer by 1/p. Also assume f(zl−1) and W are zero-centered. With that now specified, we conclude that neuron i of layer zl has the variance
Var(zli) = Var ( nin∑ k=1 W lkif(z l−1)k/p ) ≈ ninp
p2 Var(wlf(zl−1))
= E[f(zl−1)2]/p because Var(wl) = 1/nin, since we initialized W l’s columns on the unit hypersphere. Knowing this variance allows us to adjust for the influence of an arbitrary nonlinearity and a desired dropout rate.
We empirically verify that a weight initialization with this forward correction allows for consistent input distribution variance throughout the layers of a 20-layer fully connected network for differing dropout rates. The first 15 layers have 500 neurons, and the last 5 layers have 250 neurons. Specifically, we can encourage unit variance by dividing W , initialized on the unit hypersphere, by √ E(f(zl−1)2)/p. Let us compare this correction to other initializations by feeding forward a random standard normal matrix through 20 layers. Figure 1 shows the results of such an experiment, and in the experiment we use a ReLU activation function. Of course, as He initialization was designed specifically for the ReLU, it performs well when p = 1, but has an exploding distribution when there is dropout. Only the initialization with a √ E(f(zl−1)2)/p corrective term demonstrates stability when a feedforward does or does not use dropout.
2.1.2 BACKPROPAGATION
A similar analysis shows that if L is our loss function and δl = ∂L
∂zl , then
Var(δl) ≈ pnoutVar(wl+1δl+1f ′(zl)) = pE[f ′(zl)2].
In appendix A we empirically verify that this backward correction allows for consistent backpropagation error signal variance throughout the layers of a 20-layer network for differing dropout rates.
2.2 OUR INITIALIZATION
We want that Var(zl) = 1 and Var(δl) = Var(δl+1). To meet these different goals, we can initialize our weights by adding these variances, while others take the arithmetic mean of these variances or ignore the backpropagation variance altogether (He et al., 2015; Glorot & Bengio, 2010). Therefore, ifW l has its columns sampled uniformly from the surface of a unit hypersphere or is an orthonormal matrix, then our initialization is
W l/ √ E[f(zl−1)2]/p+ pE[f ′(zl)2].
For convolutional neural networks, adjusting for the backpropagation signal is less common, so one could simply use the initialization W l/ √ E[f(zl−1)2]/p. This initialization accounts for the influence of dropout rates and an arbitrary nonlinearity. We can simply initialize a random standard
Gaussian matrix and normalize its last dimension to generate W l. Another strength of this initialization is that the expectations are similar to the values in Table 1 for standardized input data, so computing mini-batch statistics is needless for our initialization (Hendrycks & Gimpel, 2016). We need only substitute in appropriate scalars during initialization. Let us now see these new adjustments in action.
2.3 EXPERIMENTS
In the experiments that follow, we utilize the MNIST dataset, a 10-class grayscale image dataset of handwritten digits with 60k training examples and 10k test examples. Then we consider CIFAR10 (Krizhevsky, 2009), a 10-class color image dataset with 50k training examples and 10k test examples. We use these data to compare our initialization with Xavier and He initializations on a fully connected neural network and with the He initialization on a convolutional neural network.
2.3.1 MNIST
Let us verify that our initialization competes with previous weight initialization schemes. To this end, we train a fully connected neural network with ReLUs, ELUs (α = 1), and the tanh activation (Clevert et al., 2016). Each 8-layer, 256 neuron wide neural network is trained for 25 epochs with a batch size of 64. From the training set, 5000 examples are held out for a validation set. With the validation set, we tune over the learning rates {10−3, 10−4, 10−5} and 7 other learning rates randomly chosen from [10−1, 10−5]. We optimize with Adam (Kingma & Ba, 2015). We also perform this task with no dropout, a dropout keep rate of 0.5, and a dropout keep rate of 0.3. Figure 2 indicates that our initialization provides faster convergence at a dropout keep rate of 0.5 for activations like the ReLU and great gains when the dropout keep rate decreases further.
2.3.2 CIFAR-10
Since VGG Net architectures (Simonyan & Zisserman, 2015) require considerable regularization and careful initialization, we use a highly regularized variant (Zagoruyko, 2015) of the architecture for our next initialization experiment. The VGG Net-like network has the stacks (2 × 3 × 64), (2 × 3 × 128), (3 × 3 × 256), (3 × 3 × 512), (3 × 3 × 512) followed by two fully-connected layers, each with 512 neurons. To regularize the deep network, we keep 70% of the neurons in the first layer, 60% in layer 3, 60% in the first two layers of the last three stacks, and 50% in the fully connected layers. Max pooling occurs after every stack, ReLU activations are applied on every neuron, and we `2 regularize with a strength of 5× 10−4. Layer width, filter count, `2 regularization strength, and dropout rate hyperparmeters are from (Zagoruyko, 2015). We compare our initialization (while deactivating the back-
propagation variance term) and the He initialization. Since the Xavier initialization is not as prominent in convolutional neural networks we do not test it. We optimize this network with two different optimizers. First, we use Nesterov momentum and tune with 10 learning rates, with 7 chosen randomly from [10−1, 10−5] and three deterministically chosen from {10−2, 10−3, 10−4}. In separate runs, we train with the Adam optimizer and tune with 10 learning rates, with 7 chosen randomly from [10−1, 10−5] and three deterministically chosen from {10−3, 10−4, 10−5}. With both optimizers we decay the learning rate by 0.1 every at the 100th and 125th epoch all while training for 150 epochs. The results in Figure 4 demonstrate the importance of small corrective dropout factors because the factors’ influence on neuron input variance changes exponentially as the network depth increases, and this can lead to vanishing update signals as shown in Figure 3. We should note that the He initialization rendered the network untrainable for more learning rates like when the learning rate was 0.01 with Nesterov momentum. However, the network converged with our initialization at this learning rate. Ultimately, our initialization provided more consistent, quick, and accurate convergence. With the Adam Optimizer, the VGG Net obtained 9.51% test set error under our initialization and 10.54% with the He initialization. Last, we use Nesterov momentum, which is a more common optimizer for deep convolutional neural networks. With Nesterov momentum, we obtained 7.41% error with our initialization and 24.65% with the He initialization.
3 BATCH NORMALIZATION VARIANCE RE-ESTIMATION
Batch Normalization aims to prevent an exploding or vanishing feedforward signal just like a good weight initialization. However, Batch Normalization has its own caveats. A practical concern,
voiced in Mishkin & Matas (2015), is that Batch Normalization can increase the feedforward time by up to 30%. Also, Ba et al. (2016) remind us that Batch Normalization cannot be applied to tasks with small batch sizes or online learning tasks lest we normalize a batch based upon mean and variance estimates from a small or single example. Batch Normalization can be used to stabilize the feedforward signal of a network with dropout, removing the need for a weight initialization which corrects for dropout variance. However, correcting for dropout’s variance is still necessary. It so happens that Batch Normalization requires that its own variance estimates be corrected before testing.
In this section we empirically show that Batch Normalization with dropout also requires special care because Batch Normalization’s estimated variance should differ between train and test. In our weight initialization derivation section 2.1, we saw that the variance of a neuron’s input grows when dropout is active. Consequently, Batch Normalization’s variance estimates are greater when dropout is active because a neuron’s input variance is greater with dropout. But dropout is deactivated during testing, and the variance estimates Batch Normalization normally uses are accurate when dropout is active not inactive. For this reason, we re-estimate the Batch Normalization variance parameters after training. We accomplish this by simply feeding forward the training data with dropout deactivated and only letting the Batch Normalization variance running averages update. When re-estimating the variance no backpropagation occurs. Let us now verify that re-estimating the Batch Normalization variance values improves test performance.
In our experiment, we turn our attention to state of the art convolutional neural networks as they use Batch Normalization and dropout. For example, Densely Connected Networks (DenseNets) (Huang et al., 2016) use dropout with Batch Normalization when training without data augmentation. Training without data augmentation is of interest because it demonstrates how data efficient an architecture is, and some images have their meaning destroyed under augmentation like mirroring (e.g., the mirror image of the digit “7” is meaningless). Moreover, Zagoruyko & Komodakis (2016) use dropout even when there is data augmentation as Batch Normalization alone does not sufficiently regularize the network. Please note that in this experiment we are only testing the effect of Batch Normalization variance adjustments, and we are not testing the effect of different weight initializations.
We turn to DenseNets in this experiment because, to our knowledge, they hold the state of the art on CIFAR-10 and CIFAR-100 without data augmentation. We train a DenseNet with dropout and Batch Normalization and re-estimate the Batch Normalization variance parameters outside of training to achieve large error reductions. These DenseNets are trained just as described in the original paper except that every 5 epochs we reset the momentum variable following a discussion with a DenseNet paper author, as this might improve accuracy. We save the DenseNet model when it has trained for half of the scheduled epochs (when it is “Halfway”) and when it is entirely done training. Then, using these models, we feed forward the training data with dropout off for one epoch
without performing any backpropagation. While the data feeds forward, we only allow the Batch Normalization moving average estimate of the variance to update. In no way does this variance re-estimation at the Halfway point affect future training because we do not train with these reestimated variance parameters. Now, DenseNets hold the state of the art on CIFAR-10 and CIFAR100 without data augmentation. Specifically, they obtain 5.77% error on CIFAR-10 without data augmentation and 23.42% on CIFAR-100 without data augmentation. Table 2 shows the results of Batch Normalization variance moving average re-estimation. The figure shows L, k, and p, which are the number of layers L, the growth factor k, and the dropout keep probability p. As an example of a table row, SVHN Original shows the error achieved in the original DenseNet paper. The row below shows the DenseNet we trained at the Halfway point (“Halfway Error”) and at the end of training (“Error”), and the error decreased under re-estimating the Batch Normalization variance. The effect of updating the variance estimation is shown under columns with “BN Update.” We see that simply feeding forward the training data without dropout and allowing the Batch Normalization variance moving averages to update lets us surpass the state of the art on CIFAR-10 and CIFAR-100 and can sometimes improve accuracy by more than 2%.
4 DISCUSSION
In practice, if we lack an estimate for a nonlinearity adjustment factor, then 0.5 is a reasonable default. A justification for a 0.5 adjustment factor comes from connections to previous weight initializations. This is because if p = 1 and we default the adjustments to 0.5, our initialization is the “Xavier” initialization if we use vectors from within the unit hypercube rather than vectors on the unit hypersphere (Glorot & Bengio, 2010). Knowing this connection, we can therefore generalize Xavier initialization to
Unif[−1, 1]× √ 3 /√ nin p E[f(zl−1)2] + pnoutE[f ′(zl)2] .
Furthermore, we can optionally exclude the backpropagation variance term—in this case, if p = 1 and f is a ReLU, our initialization is He’s initialization if we use random normal weights (He et al., 2015). Note that since He et al. (2015) considered a 0.5 corrective factor to account for the ReLU’s compressiveness (its tendency to reduce output variance), it is plausible that E[f(zl−1)2] is a general adjustment for a nonlinearity’s compressiveness. Since most neural network nonlinearities are compressive, 0.5 is a reasonable default adjustment.1 Also recall that our initialization with the backpropagation variance term amounts is
W l/ √
E[f(zl−1)2]/p+ pE[f ′(zl)2]. 1Note that the first hidden layer is adjacent to neurons which can be viewed as having an identity activation.
For these, a 1.0 factor is more appropriate, but the practical difference is miniscule.
If we use the 0.5 corrective factor default and we do not apply any dropout, then we are left with W l, an orthonormal matrix or a matrix with its columns on a unit hypersphere.
5 CONCLUSION
A simple modification to previous weight initializations shows marked improvements on fully connected and convolutional architectures. Unlike recent variance stabilization techniques, ours only relies on simple corrective factors and not special forward passes or batch statistics. For highlyregularized networks, the convergence gains are conspicuous and networks without the corrective factors are harder to train. Therefore, if a user wants to train online or not pay the computational cost Batch Normalization imposes, he or she would do well to apply a dropout corrective factor to their weight initialization matrix.
If a user is able to use Batch Normalization, the effect of dropout still cannot be ignored. The Batch Normalization variance moving averages differ between train and test, so when training is complete, we re-estimate those variance parameters. This is accomplished by feeding the training data forward for one epoch without dropout on and only allowing the variance moving averages to change. By doing so, networks can improve their accuracy notably. Indeed, by applying this simple, highly general technique, we achieved the state of the art on CIFAR-10 and CIFAR-100 without data augmentation.
ACKNOWLEDGMENTS
We would like to thank Eric Martin for numerous suggestions, Steven Basart for training the SVHN DenseNet, and our anonymous reviewers for suggestions. We would also like to thank NVIDIA Corporation for donating several TITAN X GPUs used in this research.
A A RANDOM BACKPROPAGATION
We can “feed backward” a random Gaussian matrix with standard deviation 0.01 and see how different initializations affect the distribution of error signals for each layer. Figure 5 shows the results when the backward correction factor is pE[f ′(zl)2]. Again, we used the ReLU activation function due to its widespread use, so the He initialization performs considerably well.

This paper proposes new initialization for particular architectures and a correction trick to batch normalization to correct variance introduced by dropout. While authors state interesting observations, the claims are not supported with convincing results.
I guess Figure 1 is only for mnist and for only two values of p with one particular network architecture, the dataset and empirical setup is not clear.
The convergence is demonstrated only for three dropout values in Figure 2 which may cause an unfair comparison. For instance how does the convergence compare for the best dropout rate after cross-validation (three figures each figure has three results for one method with different dropouts [bests cv result for each one])? Also how is the corresponding validation error and test iterations?  Also only mnist does not have to generalize to other benchmarks.
Figure 3 gives closer results for Adam optimizer, learning rate is not selected with random search or bayesian optimization, learning decay iterations fixed and regularization coefficient is set to a small value without tuning. A slightly better tuning of parameters may close the current gap. Also Nesterov based competitor gives unreasonably worse accuracy compared to recent results which may indicate that this experiment should not be taken into account.
In Table 2, there is no significant improvement on CIFAR10. The CIFAR100 difference is not significant without including batch normalization variance re-estimation. However there is no result for 'original with BN update' therefore it is not clear whether the BN update helps in general or not. SVHN also does not have result for original with BN update.
There should be baselines with batch normalizations for Figure 1,2 3 to support the claims convincingly.  The main criticism about batch normalization is additional computational cost by giving (Mishkin et al, 2016 ) as reference however this should not be a reason to not to compare the initialization to batch-normalization.  In fact, (Mishkin et al, 2016) performs comparison to batch normalization and also with and without data augmentation with recent state of art architectures.
None of the empirical results have data augmentation. It is not clear if the initialization or  batch normalization update will help or make it worse for that case.
Recent state of art methods methods like Res Net variant and Dense Net scale to many depths and report result for ImageNet. Although the authors claim that this can be extended to residual network variants, it is not clear if there is going to be any empirical gain for that architectures.
This work requires a comprehensive and fair comparison. Otherwise the contribution is not significant.


REVISITING CLASSIFIER TWO-SAMPLE TESTS
The goal of two-sample tests is to assess whether two samples, SP ∼ P and SQ ∼ Q, are drawn from the same distribution. Perhaps intriguingly, one relatively unexplored method to build two-sample tests is the use of binary classifiers. In particular, construct a dataset by pairing the n examples in SP with a positive label, and by pairing the m examples in SQ with a negative label. If the null hypothesis “P = Q” is true, then the classification accuracy of a binary classifier on a held-out subset of this dataset should remain near chance-level. As we will show, such Classifier Two-Sample Tests (C2ST) learn a suitable representation of the data on the fly, return test statistics in interpretable units, have a simple null distribution, and their predictive uncertainty allow to interpret where P and Q differ. The goal of this paper is to establish the properties, performance, and uses of C2ST. First, we analyze their main theoretical properties. Second, we compare their performance against a variety of state-of-the-art alternatives. Third, we propose their use to evaluate the sample quality of generative models with intractable likelihoods, such as Generative Adversarial Networks (GANs). Fourth, we showcase the novel application of GANs together with C2ST for causal discovery.
1 INTRODUCTION
One of the most fundamental problems in statistics is to assess whether two samples, SP ∼ Pn and SQ ∼ Qm, are drawn from the same probability distribution. To this end, two-sample tests (Lehmann & Romano, 2006) summarize the differences between the two samples into a real-valued test statistic, and then use the value of such statistic to accept1 or reject the null hypothesis “P = Q”. The development of powerful two-sample tests is instrumental in a myriad of applications, including the evaluation and comparison of generative models. Over the last century, statisticians have nurtured a wide variety of two-sample tests. However, most of these tests are only applicable to one-dimensional examples, require the prescription of a fixed representation of the data, return test statistics in units that are difficult to interpret, or do not explain how the two samples under comparison differ.
Intriguingly, there exists a relatively unexplored strategy to build two-sample tests that overcome the aforementioned issues: training a binary classifier to distinguish between the examples in SP and the examples in SQ. Intuitively, if P = Q, the test accuracy of such binary classifier should remain near chance-level. Otherwise, if P 6= Q and the binary classifier is able to unveil some of the distributional differences between SP and SQ, its test accuracy should depart from chance-level. As we will show, such Classifier Two-Sample Tests (C2ST) learn a suitable representation of the data on the fly, return test statistics in interpretable units, have simple asymptotic distributions, and their learned features and predictive uncertainty provide interpretation on how P and Q differ. In such a way, this work brings together the communities of statistical testing and representation learning.
The goal of this paper is to establish the theoretical properties and evaluate the practical uses of C2ST. To this end, our contributions are:
• We review the basics of two-sample tests in Section 2, as well as their common applications to measure statistical dependence and evaluate generative models. • We analyze the attractive properties of C2ST (Section 3) including an analysis of their exact
asymptotic distributions, testing power, and interpretability. 1For clarity, we abuse statistical language and write “accept” to mean “fail to reject”.
• We evaluate C2ST on a wide variety of synthetic and real data (Section 4), and compare their performance against multiple state-of-the-art alternatives. Furthermore, we provide examples to illustrate how C2ST can interpret the differences between pairs of samples. • In Section 5, we propose the use of classifier two-sample tests to evaluate the sample quality
of generative models with intractable likelihoods, such as Generative Adversarial Networks (Goodfellow et al., 2014), also known as GANs. • As a novel application of the synergy between C2ST and GANs, Section 6 proposes the use
of these methods for causal discovery.
2 TWO-SAMPLE TESTING
The goal of two-sample tests is to assess whether two samples, denoted by SP ∼ Pn and SQ ∼ Qm, are drawn from the same distribution (Lehmann & Romano, 2006). More specifically, two-sample tests either accept or reject the null hypothesis, often denoted by H0, which stands for “P = Q”. When rejectingH0, we say that the two-sample test favors the alternative hypothesis, often denoted by H1, which stands for “P 6= Q”. To accept or reject H0, two-sample tests summarize the differences between the two samples (sets of identically and independently distributed examples):
SP := {x1, . . . , xn} ∼ Pn(X) and SQ := {y1, . . . , ym} ∼ Qm(Y ) (1)
into a statistic t̂ ∈ R. Without loss of generality, we assume that the two-sample test returns a small statistic when the null hypothesis “P = Q” is true, and a large statistic otherwise. Then, for a sufficiently small statistic, the two-sample test will accept H0. Conversely, for a sufficiently large statistic, the two-sample test will reject H0 in favour of H1.
More formally, the statistician performs a two-sample test in four steps. First, decide a significance level α ∈ [0, 1], which is an input to the two-sample test. Second, compute the two-sample test statistic t̂. Third, compute the p-value p̂ = P (T ≥ t̂|H0), the probability of the two-sample test returning a statistic as large as t̂ when H0 is true. Fourth, reject H0 if p̂ < α, and accept it otherwise.
Inevitably, two-sample tests can fail in two different ways. First, to make a type-I error is to reject the null hypothesis when it is true (a “false positive”). By the definition of p-value, the probability of making a type-I error is upper-bounded by the significance level α. Second, to make a type-II error is to accept the null hypothesis when it is false (a “false negative”). We denote the probability of making a type-II error by β, and refer to the quantity π = 1− β as the power of a test. Usually, the statistician uses domain-specific knowledge to evaluate the consequences of a type-I error, and thus prescribe an appropriate significance level α. Within the prescribed significance level α, the statistician prefers the two-sample test with maximum power π.
Among others, two-sample tests serve two other uses. First, two-sample tests can measure statistical dependence (Gretton et al., 2012a). In particular, testing the independence null hypothesis “the random variables X and Y are independent” is testing the two-sample null hypothesis “P (X,Y ) = P (X)P (Y )”. In practice, the two-sample test would compare the sample S = {(xi, yi)}ni=1 ∼ P (X,Y )n to a sample Sσ = {(xi, yσ(i))}ni=1 ∼ (P (X)P (Y ))n, where σ is a random permutation of the set of indices {1, . . . , n}. This approach is consistent when considering all possible random permutations. However, since independence testing is a subset of two-sample testing, specialized independence tests may exhibit higher power for this task (Gretton et al., 2005).
Second, two-sample tests can evaluate the sample quality of generative models with intractable likelihoods, but tractable sampling procedures. Intuitively, a generative model produces good samples Ŝ = {x̂i}ni=1 if these are indistinguishable from the real data S = {xi}ni=1 that they model. Thus, the two-sample test statistic between Ŝ and S measures the fidelity of the samples Ŝ produced by the generative model. The use of two-sample tests to evaluate the sample quality of generative models include the pioneering work of Box (1980), the use of Maximum Mean Discrepancy (MMD) criterion (Bengio et al., 2013; Dziugaite et al., 2015; Lloyd & Ghahramani, 2015; Bounliphone et al., 2015; Sutherland et al., 2016), and the connections to density-ratio estimation (Kanamori et al., 2010; Wornowizki & Fried, 2016; Menon & Ong, 2016; Mohamed & Lakshminarayanan, 2016).
Over the last century, statisticians have nurtured a wide variety of two-sample tests. Classical two-sample tests include the t-test (Student, 1908), which tests for the difference in means of two
samples; the Wilcoxon-Mann-Whitney test (Wilcoxon, 1945; Mann & Whitney, 1947), which tests for the difference in rank means of two samples; and the Kolmogorov-Smirnov tests (Kolmogorov, 1933; Smirnov, 1939) and their variants (Kuiper, 1962), which test for the difference in the empirical cumulative distributions of two samples. However, these classical tests are only efficient when applied to one-dimensional data. Recently, the use of kernel methods (Smola & Schölkopf, 1998) enabled the development of two-sample tests applicable to multidimensional data. Examples of these tests include the MMD test (Gretton et al., 2012a), which looks for differences in the empirical kernel mean embeddings of two samples, and the Mean Embedding test or ME (Chwialkowski et al., 2015; Jitkrittum et al., 2016), which looks for differences in the empirical kernel mean embeddings of two samples at optimized locations. However, kernel two-sample tests require the prescription of a manually-engineered representation of the data under study, and return values in units that are difficult to interpret. Finally, only the ME test provides a mechanism to interpret how P and Q differ.
Next, we discuss a simple but relatively unexplored strategy to build two-sample tests that overcome these issues: the use of binary classifiers.
3 CLASSIFIER TWO-SAMPLE TESTS (C2ST)
Without loss of generality, we assume access to the two samples SP and SQ defined in (1), where xi, yj ∈ X , for all i = 1, . . . , n and j = 1, . . . ,m, and m = n. To test whether the null hypothesis H0 : P = Q is true, we proceed in five steps. First, construct the dataset
D = {(xi, 0)}ni=1 ∪ {(yi, 1)}ni=1 =: {(zi, li)}2ni=1. Second, shuffle D at random, and split it into the disjoint training and testing subsets Dtr and Dte, where D = Dtr ∪ Dte and nte := |Dte|. Third, train a binary classifier f : X → [0, 1] on Dtr; in the following, we assume that f(zi) is an estimate of the conditional probability distribution p(li = 1|zi). Fourth, return the classification accuracy on Dte:
t̂ = 1
nte ∑ (zi,li)∈Dte I [ I ( f(zi) > 1 2 ) = li ] (2)
as our C2ST statistic, where I is the indicator function. The intuition here is that if P = Q, the test accuracy (2) should remain near chance-level. In opposition, if P 6= Q and the binary classifier unveils distributional differences between the two samples, the test classification accuracy (2) should be greater than chance-level. Fifth, to accept or reject the null hypothesis, compute a p-value using the null distribution of the C2ST, as discussed next.
3.1 NULL AND ALTERNATIVE DISTRIBUTIONS
Each term I [I(f(zi) > 1/2) = li] appearing in (2) is an independent Bernoulli(pi) random variable, where pi is the probability of classifying correctly the example zi in Dte. First, under the null hypothesis H0 : P = Q, the samples SP ∼ Pn and SQ ∼ Qm follow the same distribution, leading to an impossible binary classification problem. In that case, ntet̂ follows a Binomial(nte, p = 12 ) distribution. Therefore, for large nte, we can use the central limit theorem to approximate the null distribution of (2) by N ( 12 , 14nte ).
Second, under the alternative hypothesis H1 : P 6= Q, the statistic ntet̂ follows a Poisson Binomial distribution, since the constituent Bernoulli random variables may not be identically distributed. In the following, we will approximate such Poisson Binomial distribution by the Binomial(n, p̄) distribution, where p̄ = 1n ∑n i=1 pi (Ehm, 1991). Therefore, we can use the central limit theorem to approximate the alternative distribution of (2) by N (p̄, p̄(1−p̄)nte ).
3.2 TESTING POWER
To analyze the power (probability of correctly rejecting false null hypothesis) of C2ST, we assume that the our classifier has an expected (unknown) accuracy of H0 : t = 12 under the null hypothesis “P = Q”, and an expected accuracy of H1 : t = 12 + under the alternative hypothesis “P 6= Q”,
where ∈ (0, 12 ) is the effect size distinguishing P from Q. Let Φ be the Normal cdf, nte the number of samples available for testing, and α the significance level. Then, Theorem 1. Given the conditions described in the previous paragraph, the approximate power of
the statistic (2) is Φ ( √ nte−Φ−1(1−α)/2√
1 4− 2
) .
See Appendix B for a proof. The power bound in Theorem 1 has an optimal order of magnitude for multi-dimensional problems (Bai & Saranadasa, 1996; Gretton et al., 2012a; Reddi et al., 2015). These are problems with fixed d and n→∞, where the power bounds do not depend on d. Remark 1. We leave for future work the study of quadratic-time C2ST with optimal power in highdimensional problems (Ramdas et al., 2015). These are problems where the ratio n/d→ c ∈ [0, 1], and the power bounds depend on d. One possible line of research in this direction is to investigate the power and asymptotic distributions of quadratic-time C2ST statistics 1nte(nte−1) ∑ i 6=j I[I(f(zi, zj) > 1 2 ) = li], where the classifier f(z, z ′) predicts if the examples (z, z′) come from the same sample.
Theorem 1 also illustrates that maximizing the power of a C2ST is a trade-off between two competing objectives: choosing a classifier that maximizes the test accuracy and maximizing the size of the test set nte. This relates to the well known bias-variance trade-off in machine learning. Indeed, simple classifiers will miss more nonlinear patterns in the data (leading to smaller test accuracy), but call for less training data (leading to larger test set sizes). On the other hand, flexible classifiers will miss less nonlinear patterns in the data (leading to higher test accuracy), but call for more training data (leading to smaller test sizes). Formally, the relationship between the test accuracy, sample size, and the flexibility of a classifier depends on capacity measures such as the VC-Dimension (Vapnik, 1998). Note that there is no restriction to perform model selection (such as cross-validation) on Dtr. Remark 2. We have focused on test statistics (2) built on top of the zero-one loss `0−1(y, y′) = I[y = y′] ∈ {0, 1}. These statistics give rise to Bernoulli random variables, which can exhibit high variance. However, our arguments are readily extended to real-valued binary classification losses. Then, the variance of such real-valued losses would describe the norm of the decision function of the classifier two-sample test, appear in the power expression from Theorem 1, and serve as a hyper-parameter to maximize power as in (Gretton et al., 2012b, Section 3).2
3.3 INTERPRETABILITY
There are three ways to interpret the result of a C2ST. First, recall that the classifier predictions f(zi) are estimates of the conditional probabilities p(li = 1|zi) for each of the samples zi in the test set. Inspecting these probabilities together with the true labels li determines which examples were correctly or wrongly labeled by the classifier, with the least or the most confidence. Therefore, the values f(zi) explain where the two distributions differ. Second, C2ST inherit the interpretability of their classifiers to explain which features are most important to distinguish distributions, in the same way as the ME test (Jitkrittum et al., 2016). Examples of interpretable features include the filters of the first layer of a neural network, the feature importance of random forests, the weights of a generalized linear model, and so on. Third, C2ST return statistics t̂ in interpretable units: these relate to the percentage of samples correctly distinguishable between the two distributions. These interpretable numbers can complement the use of p-values.
3.4 PRIOR USES
The reduction of two-sample testing to binary classification was introduced in (Friedman, 2003), studied within the context of information theory in (Pérez-Cruz, 2009; Reid & Williamson, 2011), discussed in (Fukumizu et al., 2009; Gretton et al., 2012a), and analyzed (for the case of linear discriminant analysis) in (Ramdas et al., 2016). The use of binary classifiers for two-sample testing is increasingly common in neuroscience: see (Pereira et al., 2009; Olivetti et al., 2012) and the references therein. Implicitly, binary classifiers also perform two-sample tests in algorithms that discriminate data from noise, such as unsupervised-as-supervised learning (Friedman et al., 2001), noise contrastive estimation (Gutmann & Hyvärinen, 2012), negative sampling (Mikolov et al., 2013), and GANs (Goodfellow et al., 2014).
2For a related discussion on this issue, we recommend the insightful comment by Arthur Gretton and Wittawat Jitkrittum, available at https://openreview.net/forum?id=SJkXfE5xx.
4 EXPERIMENTS ON TWO-SAMPLE TESTING
We study two variants of classifier-based two-sample tests (C2ST): one based on neural networks (C2ST-NN), and one based on k-nearest neighbours (C2ST-KNN). C2ST-NN has one hidden layer of 20 ReLU neurons, and trains for 100 epochs using the Adam optimizer (Kingma & Ba, 2015). C2ST-KNN uses k = bn1/2tr c nearest neighbours for classification. Throughout our experiments, we did not observe a significant improvement in performance when increasing the flexibility of these classifiers (e.g., increasing the number of hidden neurons or decreasing the number of nearest neighbors). When analyzing one-dimensional data, we compare the performance of C2ST-NN and C2ST-KNN against the Wilcoxon-Mann-Whitney test (Wilcoxon, 1945; Mann & Whitney, 1947), the Kolmogorov-Smirnov test (Kolmogorov, 1933; Smirnov, 1939), and the Kuiper test (Kuiper, 1962). In all cases, we also compare the performance of C2ST-NN and C2ST-KNN against the linear-time estimate of the Maximum Mean Discrepancy (MMD) criterion (Gretton et al., 2012a), the ME test (Jitkrittum et al., 2016), and the SCF test (Jitkrittum et al., 2016). We use a significance level α = 0.05 across all experiments and tests, unless stated otherwise. We use Gaussian approximations to compute the null distributions of C2ST-NN and C2ST-KNN. We use the implementations of the MMD, ME, and SCF tests gracefully provided by Jitkrittum et al. (2016), the scikit-learn implementation of the Kolmogorov-Smirnov and Wilcoxon tests, and the implementation from https://github.com/aarchiba/kuiper of the Kuiper test. The implementation of our experiments is available at https://github.com/lopezpaz/classifier_tests.
4.1 EXPERIMENTS ON TWO-SAMPLE TESTING
Control of type-I errors We start by evaluating the correctness of all the considered two-sample tests by examining if the prescribed significance level α = 0.05 upper-bounds their type-I error. To do so, we draw x1, . . . , xn, y1, . . . , yn ∼ N (0, 1), and run each two-sample test on the two samples {xi}ni=1 and {yi}ni=1. In this setup, a type-I error would be to reject the true null hypothesis. Figure 1(a) shows that the type-I error of all tests is upper-bounded by the prescribed significance level, for all n ∈ {25, 50, 100, 500, 1000, 5000, 10000} and 100 random repetitions. Thus, all tests control their type-I error as expected, up to random variations due to finite experiments.
Gaussian versus Student We consider distinguishing between samples drawn from a Normal distribution and samples drawn from a Student’s t-distribution with ν degrees of freedom. We shift and scale both samples to exhibit zero-mean and unit-variance. Since the Student’s t distribution approaches the Normal distribution as ν increases, a two-sample test must focus on the peaks of the distributions to distinguish one from another. Figure 1(b,c) shows the percentage of type-II errors made by all tests as we vary separately n and ν, over 100 trials (random samples). We set
n = 2000 when ν varies, and let ν = 3 when n varies. The Wilcoxon-Mann-Whitney exhibits the worst performance, as expected (since the ranks mean of the Gaussian and Student’s t distributions coincide) in this experiment. The best performing method is the the one-dimensional Kuiper test, followed closely by the multi-dimensional tests C2ST-NN and ME.
Independence testing on sinusoids For completeness, we showcase the use two-sample tests to measure statistical dependence. This can be done, as described in Section 2, by performing a two-sample test between the observed data {(xi, yi)}ni=1 and {(xi, yσ(i))}ni=1, where σ is a random permutation. Since the distributions P (X)P (Y ) and P (X,Y ) are bivariate, only the C2ST-NN, C2ST-KNN, MMD, and ME tests compete in this task. We draw (xi, yi) according to the generative model xi ∼ N (0, 1), i ∼ N (0, γ2), and yi ∼ cos(δxi) + i. Here, xi are iid examples from the random variable X , and yi are iid examples from the random variable Y . Thus, the statistical dependence between X and Y weakens as we increase the frequency δ of the sinusoid, or increase the variance γ2 of the additive noise. Figure 1(d,e,f) shows the percentage of type-II errors made by C2ST-NN, C2ST-KNN, MMD, and ME as we vary separately n, δ, and γ over 100 trials. We let n = 2000, δ = 1, γ = 0.25 when fixed. Figure 1(d,e,f) reveals that among all tests, C2ST-NN is the most efficient in terms of sample size, C2ST-KNN is the most robust with respect to high-frequency variations, and that C2ST-NN and ME are the most robust with respect to additive noise.
Distinguishing between NIPS articles We consider the problem of distinguishing between some of the categories of the 5903 articles published in the Neural Information Processing Systems (NIPS) conference from 1988 to 2015, as discussed in Jitkrittum et al. (2016). We consider articles on Bayesian inference (Bayes), neuroscience (Neuro), deep learning (Deep), and statistical learning theory (Learn). Table 1 shows the type-I errors (Bayes-Bayes row) and powers (rest of rows) for the tests reported in (Jitkrittum et al., 2016), together with C2ST-NN, at a significance level α = 0.01, when averaged over 500 trials. In these experiments, C2ST-NN achieves maximum power, while upper-bounding its type-I error by α.
Distinguishing between facial expressions Finally, we apply C2ST-NN to the problem of distinguishing between positive (happy, neutral, surprised) and negative (afraid, angry, disgusted) facial expressions from the Karolinska Directed Emotional Faces dataset, as discussed in (Jitkrittum et al., 2016). See the fourth plot of Figure 2, first two-rows, for one example of each of these six emotions. Table 2 shows the type-I errors (± vs ± row) and the powers (+ vs − row) for the tests reported in (Jitkrittum et al., 2016), together with C2ST-NN, at α = 0.01, averaged over 500 trials. C2ST-NN achieves a near-optimal power, only marginally behind the perfect results of SCF-full and MMD-quad.
5 EXPERIMENTS ON GENERATIVE ADVERSARIAL NETWORK EVALUATION
Since effective generative models will produce examples barely distinguishable from real data, twosample tests arise as a natural alternative to evaluate generative models. Particularly, our interest is to evaluate the sample quality of generative models with intractable likelihoods, such as GANs (Goodfellow et al., 2014). GANs implement the adversarial game
min g max d E x∼P (X) [log(d(x))] + E z∼P (Z) [log(1− d(g(z)))] , (3)
where d(x) depicts the probability of the example x following the data distribution P (X) versus being synthesized by the generator. This is according to a trainable discriminator function d. In the adversarial game, the generator g plays to fool the discriminator d by transforming noise vectors z ∼ P (Z) into real-looking examples g(z). On the opposite side, the discriminator plays to distinguish between real examples x and synthesized examples g(z). To approximate the solution to (3), alternate the optimization of the two losses (Goodfellow et al., 2014) given by
Ld(d) = Ex [`(d(x), 1)] + Ez [`(d(g(z)), 0)] , Lg(g) = Ex [`(d(x), 0)] + Ez [`(d(g(z)), 1)] . (4)
Under the formalization (4), the adversarial game reduces to the sequential minimization of Ld(d) and Lg(g), and reveals the true goal of the discriminator: to be the C2ST that best distinguishes data examples x ∼ P and synthesized examples x̂ ∼ P̂ , where P̂ is the probability distribution induced by sampling z ∼ P (Z) and computing x̂ = g(z). The formalization (4) unveils the existence of an arbitrary binary classification loss function ` (See Remark 2), which in turn decides the divergence minimized between the real and fake data distributions (Nowozin et al., 2016).
Unfortunately, the evaluation of the log-likelihood of a GANs is intractable. Therefore, we will employ a two-sample test to evaluate the quality of the fake examples x̂ = g(z). In simple terms, evaluating a GAN in this manner amounts to withhold some real data from the training process, and use it later in a two-sample test against the same amount of synthesized data. When the twosample test is a binary classifier (as discussed in Section 3), this procedure is simply training a fresh discriminator on a fresh set of data. Since we train and test this fresh discriminator on held-out examples, it may differ from the discriminator trained along the GAN. In particular, the discriminator trained along with the GAN may have over-fitted to particular artifacts produced by the generator, thus becoming a poor C2ST.
We evaluate the use of two-sample tests for model selection in GANs. To this end, we train a number of DCGANs (Radford et al., 2016) on the bedroom class of LSUN (Yu et al., 2015) and the Labeled Faces in the Wild (LFW) dataset (Huang et al., 2007). We reused the Torch7 code of Radford et al. (2016) to train a set of DCGANs for {1, 10, 50, 100, 200} epochs, where the generator and discriminator networks are convolutional neural networks (LeCun et al., 1998) with {1, 2, 4, 8} × gf and {1, 2, 4, 8} × df filters per layer, respectively. We evaluate each DCGAN on 10, 000 held-out examples using the fastest multi-dimensional two-sample tests: MMD, C2ST-NN, and C2ST-KNN.
Our first experiments revealed an interesting result. When performing two-sample tests directly on pixels, all tests obtain near-perfect test accuracy when distinguishing between real and synthesized (fake) examples. Such near-perfect accuracy happens consistently across DCGANs, regardless of the visual quality of their examples. This is because, albeit visually appealing, the fake examples contain checkerboard-like artifacts that are sufficient for the tests to consistently differentiate between real and fake examples. Odena et al. (2016) discovered this phenomenon concurrently with us.
On a second series of experiments, we featurize all images (both real and fake) using a deep convolutional ResNet (He et al., 2015) pre-trained on ImageNet, a large dataset of natural images (Russakovsky et al., 2015). In particular, we use the resnet-34 model from Gross & Wilber (2016). Reusing a model pre-trained on natural images ensures that the test will distinguish between real and fake examples based only on natural image statistics, such as Gabor filters, edge detectors, and so on. Such a strategy is similar to perceptual losses (Johnson et al., 2016) and inception scores (Salimans et al., 2016). In short, in order to evaluate how natural the images synthesized by a DCGAN look, one must employ a “natural discriminator”. Table 3 shows three GANs producing poor samples and three GANs producing good samples for the LSUN and LFW datasets, according to the MMD, C2ST-KNN, C2ST-NN tests on top of ResNet features. See Appendix A for the full list of results. Although it is challenging to provide with an objective evaluation of our results, we believe that the rankings provided by two-sample tests could serve for efficient early stopping and model selection. Remark 3 (How good is my GAN? Is it overfitting?). Evaluating generative models is a delicate issue (Theis et al., 2016), but two-sample tests may offer some guidance. In particular, good (nonoverfitting) generative models should produce similar two-sample test statistics when comparing their generated samples to both the train-set and the test-set samples. 3 As a general recipe, prefer generative models that achieve the same and small two-sample test statistic when comparing their generated samples to both the train-set and test-set samples.
5.1 EXPERIMENTS ON INTERPRETABILITY
We illustrate the interpretability power of C2ST. First, the predictive uncertainty of C2ST sheds light on where the two samples under consideration agree or differ. In the first plot of Figure 2, a C2ST-NN separates two bivariate Gaussian distributions with different means. When performing this separation, the C2ST-NN provides an explicit decision boundary that illustrates where the two distributions separate from each other. In the second plot of Figure 2, a C2ST-NN separates a Gaussian distribution from a Student’s t distribution with ν = 3, after scaling both to zero-mean and unit-variance. The plot reveals that the peaks of the distributions are their most differentiating feature. Finally, the third plot of Figure 2 displays, for the LFW and LSUN datasets, five examples classified as real with high uncertainty (first row, better looking examples), and five examples classified as fake with high certainty (second row, worse looking examples).
Second, the features learnt by the classifier of a C2ST are also a mechanism to understand the differences between the two samples under study. The third plot of Figure 2 shows six examples from the Karolinska Directed Emotional Faces dataset, analyzed in Section 4.1. In that same figure, we arrange the weights of the first linear layer of C2ST-NN into the feature most activated at positive examples (bottom left, positive facial expressions), the feature most activated at negative examples (bottom middle, negative facial expressions), and the “discriminative feature”, obtained by substracting these two features (bottom right). The discriminative feature of C2ST-NN agrees with the one found by (Jitkrittum et al., 2016): positive and negative facial expressions are best distinguished at the eyebrows, smile lines, and lips. A similar analysis Jitkrittum et al. (2016) on the C2ST-NN features in the NIPS article classification problem (Section 4.1) reveals that the features most activated for the “statistical learning theory” category are those associated to the words inequ, tight, power, sign, hypothesi, norm, hilbert. The features most activated for the “Bayesian inference” category are those associated to the words infer, markov, graphic, conjug, carlo, automat, laplac.
6 EXPERIMENTS ON CONDITIONAL GANS FOR CAUSAL DISCOVERY
In causal discovery, we study the causal structure underlying a set of d random variables X1, . . . , Xd. In particular, we assume that the random variables X1, . . . , Xd share a causal structure described by a collection of Structural Equations, or SEs (Pearl, 2009). More specifically, we assume that the random variable Xi takes values as described by the SE Xi = gi(Pa(Xi,G), Ni), for all i = 1, . . . , d. In the previous, G is a Directed Acyclic Graph (DAG) with vertices associated to each of the random variables X1, . . . , Xd. Also in the same equation, Pa(Xi,G) denotes the set of random variables which are parents of Xi in the graph G, and Ni is an independent noise random variable that follows
3As discussed with Arthur Gretton, if the generative model memorizes the train-set samples, a sufficiently large set of generated samples would reveal such memorization to the two-sample test. This is because some unique samples would appear multiple times in the set of generated samples, but not in the test-set of samples.
the probability distribution P (Ni). Then, we say that Xi → Xj if Xi ∈ Pa(Xj), since a change in Xi will cause a change in Xj , as described by the i-th SE.
The goal of causal discovery is to infer the causal graph G given a sample from P (X1, . . . , Xd). For the sake of simplicity, we focus on the discovery of causal relations between two random variables, denoted by X and Y . That is, given the sample D = {(xi, yi)}ni=1 ∼ Pn(X,Y ), our goal is to conclude whether “X causes Y ”, or “Y causes X”. We call this problem cause-effect discovery (Mooij et al., 2016). In the case where X → Y , we can write the cause-effect relationship as:
x ∼ P (X), n ∼ P (N), y ← g(x, n). (5) The current state-of-the-art in the cause-effect discovery is the family of Additive Noise Models, or ANM (Mooij et al., 2016). These methods assume that the SE (5) allow the expression y ← g(x) +n, and exploit the independence assumption between the cause random variable X and the noise random variable N to analyze the distribution of nonlinear regression residuals, in both causal directions.
Unfortunately, assuming independent additive noise is often too simplistic (for instance, the noise could be heteroskedastic or multiplicative). Because of this reason, we propose to use Conditional Generative Adversarial Networks, or CGANs (Mirza & Osindero, 2014) to address the problem of cause-effect discovery. Our motivation is the shocking resemblance between the generator of a CGAN and the SE (5): the random variable X is the conditioning variable input to the generator, the random variable N is the noise variable input to the generator, and the random variable Y is the variable synthesized by the generator. Furthermore, CGANs respect the independence between the cause X and the noise N by construction, since n ∼ P (N) is independent from all other variables. This way, CGANs bypass the additive noise assumption naturally, and allow arbitrary interactions g(X,N) between the cause variable X and the noise variable N .
To implement our cause-effect inference algorithm in practice, recall that training a CGAN from X to Y minimizes the two following objectives in alternation:
Ld(d) = Ex,y [`(d(x, y), 1)] + Ex,z [`(d(x, g(x, z)), 0)] , Lg(g) = Ex,y [`(d(x, y), 0)] + Ex,z [`(d(x, g(x, z)), 1)] .
Our recipe for cause-effect is to learn two CGANs: one with a generator gy fromX to Y to synthesize the dataset DX→Y = {(xi, gy(xi, zi))}ni=1, and one with a generator gx from Y to X to synthesize the dataset DX←Y = {(gx(yi, zi), yi)}ni=1. Then, we prefer the causal direction X → Y if the two-sample test statistic between the real sample D and DX→Y is smaller than the one between D and DY→X . Thus, our method is Occam’s razor at play: declare the simplest direction (in terms of conditional generative modeling) as the true causal direction.
Table 4 summarizes the performance of this procedure when applied to the 99 Tübingen cause-effect pairs dataset, version August 2016 (Mooij et al., 2016). RCC is the Randomized Causation Coefficient of (Lopez-Paz et al., 2015). The Ensemble-CGAN-C2ST trains 100 CGANs, and decides the causal
direction by comparing the top generator obtained in each causal direction, as told by C2ST-KNN. The need to ensemble is a remainder of the unstable behaviour of generative adversarial training, but also highlights the promise of such models for causal discovery.
7 CONCLUSION
Our take-home message is that modern binary classifiers can be easily turned into powerful two-sample tests. We have shown that these classifier two-sample tests set a new state-of-the-art in performance, and enjoy unique attractive properties: they are easy to implement, learn a representation of the data on the fly, have simple asymptotic distributions, and allow different ways to interpret how the two samples under study differ. Looking into the future, the use of binary classifiers as two-sample tests provides a flexible and scalable approach for the evaluation and comparison of generative models (such as GANs), and opens the door to novel applications of these methods, such as causal discovery.
96 96 200 0.849 0.999 1.000
96 96 100 0.668 0.999 1.000
96 96 50 0.829 0.999 1.000
96 96 10 0.671 0.999 1.000
96 96 1 0.672 0.999 1.000
96 64 200 0.020 0.711 0.972
96 64 50 0.017 0.812 0.962
96 64 1 0.287 0.978 1.000
96 32 200 0.024 0.689 0.981
96 32 100 0.022 0.745 0.965
96 32 10 0.022 0.770 0.991
96 32 1 0.356 0.986 1.000
64 96 200 0.020 0.755 0.983
64 96 100 0.016 0.866 0.976
64 96 50 0.015 0.801 0.980
64 96 10 0.158 0.830 0.999
64 96 1 0.891 0.996 1.000
64 64 200 0.021 0.775 0.980
64 64 100 0.019 0.685 0.978
64 64 50 0.014 0.864 0.978
64 64 10 0.021 0.825 0.988
64 64 1 0.313 0.964 1.000
64 32 200 0.018 0.736 0.963
64 32 100 0.017 0.753 0.959
64 32 50 0.018 0.788 0.977
64 32 10 0.023 0.842 0.979
64 32 1 0.389 0.987 1.000
32 96 200 0.928 0.991 1.000
32 96 100 0.928 0.991 1.000
32 96 50 0.924 0.991 1.000
32 96 10 0.927 0.991 1.000
32 96 1 0.915 0.997 1.000
32 64 200 0.048 0.962 1.000
32 64 100 0.032 0.936 0.998
32 64 50 0.020 0.804 0.989
32 64 10 0.035 0.897 0.997
32 64 1 0.330 0.984 1.000
32 32 100 0.014 0.797 0.974
32 32 50 0.026 0.758 0.983
32 32 10 0.024 0.831 0.996
32 32 1 0.154 0.994 1.000
A RESULTS ON EVALUATION OF GENERATIVE ADVERSARIAL NETWORKS
96 96 200 0.923 1.000 1.000
96 96 100 0.920 1.000 1.000
96 96 50 0.926 1.000 1.000
96 96 10 0.920 1.000 1.000
96 96 1 0.969 1.000 1.000
96 64 200 0.020 0.697 0.976
96 64 100 0.028 0.744 0.983
96 64 10 0.042 0.904 0.999
96 64 1 0.715 1.000 1.000
96 32 200 0.021 0.780 0.973
96 32 100 0.016 0.735 0.964
96 32 50 0.026 0.734 0.965
96 32 10 0.222 0.978 1.000
96 32 1 1.022 1.000 1.000
64 96 200 1.061 1.000 1.000
64 96 100 1.063 1.000 1.000
64 96 50 1.002 1.000 1.000
64 96 10 1.015 1.000 1.000
64 96 1 1.003 1.000 1.000
64 64 200 0.022 0.719 0.974
64 64 50 0.018 0.712 0.973
64 64 10 0.103 0.910 0.998
64 64 1 0.836 1.000 1.000
64 32 200 0.025 0.757 0.972
64 32 100 0.018 0.746 0.973
64 32 50 0.043 0.756 0.988
64 32 10 0.086 0.971 1.000
64 32 1 1.041 1.000 1.000
64 96 200 0.037 0.779 0.995
32 96 100 0.053 0.778 0.987
32 96 50 0.995 1.000 1.000
32 96 10 0.992 1.000 1.000
32 96 1 0.995 1.000 1.000
32 64 100 0.036 0.766 0.972
32 64 50 0.050 0.808 0.985
32 64 10 0.989 1.000 1.000
32 64 1 0.994 1.000 1.000
32 32 200 0.022 0.776 0.970
32 32 100 0.029 0.808 0.982
32 32 50 0.042 0.788 0.993
32 32 10 0.152 0.940 1.000
32 32 1 0.806 1.000 1.000
B PROOF OF THEOREM 1
Our statistic is a random variable T ∼ N (
1 2 , 1 4nte
) under the null hypothesis, and T ∼
N (
1 2 + , n −1 te ( 1 4 − 2 )) under the alternative hypothesis. Furthermore, at a significance level
α, the threshold of our statistic is zα = 12 + Φ−1(1−α)√ 4nte ; under this threshold we would accept the null hypothesis. Then, the probability of making a type-II error is
P T∼N ( 1 2 + , 1 4 − 2
nte
) (T < zα) = P T ′∼N ( 0, 1 4 − 2
nte
)(T ′ < Φ−1(1− α)√ 4nte − )
= Φ
(√ nte
1 4 − 2
( Φ−1(1− α)√ 4nte − ))
= Φ Φ−1(1− α)/2− √nte√ 1 4 − 2  . Therefore, the power of the test is
π(α, nte, ) = 1− Φ Φ−1(1− α)/2− √nte√ 1 4 − 2  = Φ  √nte − Φ−1(1− α)/2√ 1 4 − 2  , which concludes the proof.
C ACKNOWLEDGEMENTS
We are thankful to L. Bottou, B. Graham, D. Kiela, M. Rojas-Carulla, I. Tolstikhin, and M. Tygert for their help in improving the quality of this manuscript. This work was partly supported by ERC grant LEAP (no. 336845) and CIFAR Learning in Machines & Brains program.

## Paper summary
The paper reconsiders the idea of using a binary classifier to do two-sample testing. The idea is to split the sample into two disjoint training and test sets, train a classifier on the training set, and use the accuracy on the test set as the test statistic. If the accuracy is above chance level, one concludes that the two samples are from different distributions i.e., reject H0.
A theoretical result on an asymptotic approximate test power is provided. One implication is that the test is consistent, assuming that the classifier is better than coin tossing. Experiments on toy problems, evaluation of GANs, and causal discovery verify the effectiveness of the test. In addition, when the classifier is a neural net, examining the first linear filter layer allows one to see features which are most activated. The result is an interpretable visual indicator of how the two samples differ.
## Review summary
The paper is well written and easy to follow. The idea of using a binary classifier for a two-sample testing is not new, as made clear in the paper. The main contributions are the analysis of the asymptotic test power, the use of modern deep nets as the classifier in this context, and the empirical studies on various tasks. The empirical results are satisfactorily convincing.  Although not much discussion is made on why the method works well in practice, overall contributions have a potential to start a new direction of research on model criticisms of generative models, as well as visualization of where a model fails. I vote for an acceptance.
## Major comments / questions
My main concern is on Theorem 1 (asymptotic test power) and its assumptions.  But, I understand that these can be fixed as discussed below.
* Under H0, the distribution of the test statistic (i.e., sum of 0-1 classification results) follows Binomial(nte, 1/2) as stated.  However, under H1, terms in the sum are independent but *not* identical Bernoulli random variable. This is because each term depends on a data point z_i, which can be from either P or Q. So, in the paragraph in Sec3.1: "... the random variable n_te \hat{t} follows a Binomial(nte, p)..." is not correct. Essentially p depends on z_i. It should follow a Poisson binomial distribution.
* In the same paragraph, for the same reason, the alternative distribution of Binomial(nte, p=p_{risk}) is probably not correct. I guess you mention it to use Moivre-Laplace to get the asymptotic normality.
Anyway, I see no reason why you would need this statement as the Binomial is not required in the proof, but only its asymptotic normality. A variant of the central limit theorem (instead of the Moivre-Laplace theorem) for independent, non-identical variables would still allow you to conclude the asymptotic normality of the Poisson binomial (with some conditions). See for example