Fast Two-Sample Testing with Analytic Representations of Probability Measures
We propose a class of nonparametric two-sample tests with a cost linear in the sample size. Two tests are given, both based on an ensemble of distances between analytic functions representing each of the distributions. The first test uses smoothed empirical characteristic functions to represent the distributions, the second uses distribution embeddings in a reproducing kernel Hilbert space. Analyticity implies that differences in the distributions may be detected almost surely at a finite number of randomly chosen locations/frequencies. The new tests are consistent against a larger class of alternatives than the previous linear-time tests based on the (non-smoothed) empirical characteristic functions, while being much faster than the current state-of-the-art quadratic-time kernel-based or energy distancebased tests. Experiments on artificial benchmarks and on challenging real-world testing problems demonstrate that our tests give a better power/time tradeoff than competing approaches, and in some cases, better outright power than even the most expensive quadratic-time tests. This performance advantage is retained even in high dimensions, and in cases where the difference in distributions is not observable with low order statistics.
Summary: This paper proposes two fast methods for nonparametric two sample testing. These are based on the notion of random metrics on distributions. The authors propose two approaches to make random metrics: one based on smooth characteristic functions, and one based of kernel mean embeddings with analytic kernels. Both the approaches essentially make use of representations of probabilities as analytic functions (in an RKHS). Analytic functions are different from each other almost surely w.r.t. any absolutely continuous distribution. Therefore almost surely unique representations of probabilities can be defined, by evaluating of the analytic representations of those probabilities on a random sample from an absolutely continuous distribution. By doing such evaluations on J random samples, the probabilities can be a.s. uniquely represented as J dimensional vectors. The random metrics are defined as distances between these J dimensional vectors. The resulting tests are fast to compute, and have high statistical powers.I think this paper is nice and should be accepted. It is clever to represent distributions by finite dimensional vectors with analytic functions. I believe this approach can also speed up other methods based on kernel mean embeddings, and therefore would have significant impacts. Quality: Both the theoretical analysis and experiments of the paper are well conducted. The following are some comments.As J goes to infinity, Eq. (10) tends to || \mu_P - \mu_Q ||_{L2(\nu)}, where \nu is the absolutely continuous distribution of T_j. This further equals to || T^{1/2} (\mu_P - \mu_Q) ||_H, where T denotes the integral operator defined by the kernel and the distribution \nu. This can be seen as a smoothed version of MMD, as T^{1/2} has an effect of smoothing. Therefore Eq. (10) may be seen as an estimator of the smoothed MMD. This might explain why the test based on mean embedding performs well empirically for high-dimensional settings. Is this understanding correct? And if the space of the rebuttal permits, please explain how the performance change when J is increased. Clarity: This is paper is well organized and easy to understand.Originality: I believe that clever idea of using analytic functions to represent distributions is original. Significance: The paper proposes approaches for representing distributions a.s. uniquely as finite dimensional vectors. I believe these are beneficial not only to two sample testing, but also to other problems of nonparametric statics. Examples include independence testing, Bayesian inference with graphical models, and reinforcement learning. These have been effectively tackled with kernel mean embeddings, but they are known to be slow. The approaches proposed in this paper can be used to speed up those kernel methods. Therefore I believe this paper would have significant impacts. Minor points: Line 164: if P \neq Q then d(P,Q) > 0 a.s. ?