A Linear-Time Kernel Goodness-of-Fit Test
We propose a novel adaptive test of goodness-of-fit, with computational cost linear in the number of samples. We learn the test features that best indicate the differences between observed samples and a reference model, by minimizing the false negative rate. These features are constructed via Stein's method, meaning that it is not necessary to compute the normalising constant of the model. We analyse the asymptotic Bahadur efficiency of the new test, and prove that under a mean-shift alternative, our test always has greater relative efficiency than a previous linear-time kernel test, regardless of the choice of parameters for that test. In experiments, the performance of our method exceeds that of the earlier linear-time test, and matches or exceeds the power of a quadratic-time kernel test. In high dimensions and where model structure may be exploited, our goodness of fit test performs far better than a quadratic-time two-sample test based on the Maximum Mean Discrepancy, with samples drawn from the model.
The kernel Stein discrepancy goodness-of-fit test is based on a kernelized Stein operator T_p such that under the distribution p (and only under the distribution p), (T_p f) has mean 0 for all test functions f in an RKHS unit ball. The maximum mean of (T_p f) over the class of test functions under a distribution q thus provides a measure of the discrepancy between q and p. This quantity is known as the kernel Stein discrepancy and can also be expressed as the RKHS norm of a witness function g.
In this paper, instead of estimating the RKHS norm of g using a full-sample U-statistic (which requires quadratic time to compute) or an incomplete U-statistic (which takes linear time but suffers from low power), the authors compute the empirical norm of g at a finite set of locations, which are either generated from a multivariate normal distribution fitted to the data or are chosen to approximately maximize test power. This produces a linear-time test statistic whose power is comparable to that of the quadratic-time kernel Stein discrepancy in simulations. The authors also analyze the test's approximate Bahadur efficiency.
This paper is well-written and well-organized, and the proposed test dramatically improves upon the computational burden of other tests, such as the maximum mean discrepancy and kernel Stein discrepancy tests, without sacrificing too much power in simulations. I certainly believe this paper is worthy of inclusion in NIPS.
Although this is not often a salient concern in machine learning, in more traditional statistical applications of hypothesis testing, randomized tests engender quite a bit of nervousness; we don't want a rejection decision in a costly clinical trial or program evaluation to hinge on having gotten a lucky RNG seed. For permutation tests or bootstrapped test statistics, we know that the rejection decision is stable as long as enough permutations or bootstrap iterations are performed. For the authors' proposed test, on the other hand, randomization is involved in choosing the test locations v_1 through v_J (or in choosing the initial locations for gradient ascent), and the simulations use a very small value of J (namely, J=5). Furthermore, in order to estimate parameters of the null distribution of the test statistic, a train-test split is needed. Which raises the question: are the test statistic and rejection threshold stable (for fixed data, across different randomizations)? It would be helpful to have additional simulations regarding the stability of the rejection decision for a fixed set of data, and guidelines as to how many train-test splits are required / how large J needs to be to achieve stability.
A minor comment: line 70, switch the order of "if and only if" and "for all f".