A Wild Bootstrap for Degenerate Kernel Tests
A wild bootstrap method for nonparametric hypothesis tests based on kernel distribution embeddings is proposed. This bootstrap method is used to construct provably consistent tests that apply to random processes, for which the naive permutation-based bootstrap fails. It applies to a large group of kernel tests based on V-statistics, which are degenerate under the null hypothesis, and nondegenerate elsewhere. To illustrate this approach, we construct a two-sample test, an instantaneous independence test and a multiple lag independence test for time series. In experiments, the wild bootstrap gives strong performance on synthetic examples, on audio data, and in performance benchmarking for the Gibbs sampler. The code is available at https://github.com/kacperChwialkowski/ wildBootstrap.
In this paper, which is in the general context of nonparametric testing based on Hilbert space embeddings, a novel Type 1 error control calibration method for nonparametric hypothesis tests of two weakly dependent samples/processes is proposed. The paper strongly builds on recent work in mathematical statistics [18,25] regarding the wild bootstrap method. The main contribution consists in showing that V statistics (of any order) and in particular those based on RKHS embeddings (including MMD and HSIC) can be asymptotically consistent even for weakly dependent processes (tau mixing). The approach is evaluated in terms of the achieved Type 1 and 2 error rates on syntethic, audio, and Gibbs sampler data.First of all, the paper is very well written and concise, and despite the high complexity, relatively easy to follow. The paper addresses a the very relevant topic of nonparametric statistical testing based on RKHS embeddings of probability distributions, which has been a vivid topic in ML in recent years. The main contributions of the paper from my point of view are:- introducing the wild bootstrap method in ML (has solely been used in statistical literature up to now)- proving that the wild bootstrap yields asymptotic consistency under weak dependence assumptions (tau mixing), for both the null and alternative hypotheses- Indicating the empirical usefulnesssThe strongest parts of the paper are the methodological contribution and innovative idea, and of course the strong theoretical part, establishing deep theoretical foundations of the suggested testing procedure. The analysis, which is very sounds, is general enough to apply also to V statistics beyond the common ones in ML (i.e., beyond HSIC and MMD). The theory alone would make a good NIPS contribution, but in addition there is also a empirical analysis, which is -- although not entirely convincing when being picky (see detailed comments below) -- promising and shows superior performance of the method in comparison to appropriate baselines.In summary, this is a novel methodology for a relevant NIPS topic, which is addressed in deep theoretical detail, and included a promising empirical analysis. A clear accept.Minor, quite picky comments:- Introduce probability operator \Epsilon or use standard notation like \mathbb P- Use of the word "control": please correct me if I am wrong, but in my understanding the type 1 error is controlled if the truly measured type 1 error is *smaller or equal* to the desired level. In several experiments (e.g., MCMC Vb1) the actual rate is slighly higher than the desired one, which would strikly speaking mean the error is not entirely controlled  (if I get the terminology right).-  Introduction of the wild bootstrap at the end of Sect 2: this is very technical, but nobody in ML knows this quantity. Show an illustration, what does it do intuitively? Example of how to choose the wild bootstrap process: for instance, the statistical learning theory reader might wonder whether it makes sense to use a correlated Rademacher or Gaussian process here.- Beginning of Sec 4: "This quantity". Let's say I wouldn't know anything about MMD: it is unclear to which term "this quantity" refers.- Top of page 5: "[18] and 2" -> [18] and [25] ?- experiments: make clearer what the difference between MMD hat and Vb1 and Vb2 is. Also note that the Type 1 error is not controlled, it is just closer to the desired error- For a sound empirical analysis, you could use ROC curves. It is neccessary to compare the type 1 error and the power / type 2 error at various level simultaeoulsy for a fair comparision. So:1. at some places in the text (e.g., end of page 7) you mention the measured type 2 error. This is important and I would suggest mentioning this also in the captions of the figures.2. What does Figure 1 tell me? The methods could be better evaluated if you plotted a roc or pr curve or something like that, I would think.3. For the same reason: figure 2 and end of page 8. "was around 5%". This is a little vague and again this should be mentioned in the caption of figure 2.Same page: "The type... was equal or lower ..." This is again vague. A roc or pr curve would shed light on this.- The Bonferoni correction for HSIC-Lag will be too conservative under the alternative hypothesis. Is there a way to wrap a (wild) bootstrap around the whole lagged kind of procedure / come up with a joint statistics, which is calibrated via wild bootstrap?