Robust Hypothesis Test for Nonlinear Effect with Gaussian Processes
This work constructs a hypothesis test for detecting whether an data-generating function h : R p → R belongs to a specific reproducing kernel Hilbert space H 0 , where the structure of H 0 is only partially known. Utilizing the theory of reproducing kernels, we reduce this hypothesis to a simple one-sided score test for a scalar parameter, develop a testing procedure that is robust against the misspecification of kernel functions, and also propose an ensemble-based estimator for the null model to guarantee test performance in small samples. To demonstrate the utility of the proposed method, we apply our test to the problem of detecting nonlinear interaction between groups of continuous features. We evaluate the finite-sample performance of our test under different data-generating functions and estimation strategies for the null model. Our results reveal interesting connections between notions in machine learning (model underfit/overfit) and those in statistical inference (i.e. Type I error/power of hypothesis test), and also highlight unexpected consequences of common model estimating strategies (e.g. estimating kernel hyperparameters using maximum likelihood estimation) on model inference.
This paper considers an interesting issue of hypothesis testing with complex black models. I will say from the outset that I am skeptical of the NHST framework, especially as it is currently applied in classical statistical settings, where I think it is clear the ongoing harm that it has done to applied science (see, e.g. the ASA's statement on p-values). Whether I should be more or less skeptical of it in nonparametric settings is an interesting questions and I'd be happy to be convinced otherwise. I guess the standard arguments aren't worth rehashing in great detail but just so you're aware of where I'm coming from, if I care about the (possibly non-linear) interaction effect then I want a kernel parameterized as:
beta1 * k1a(x1,x1') + beta2 * k2c(x2,x2') + beta12 * k1b(x1,x1')*k2b(x2,x2')
I want to fit that model using all of my expensive data, and then say something about beta12. Bayesian methods would give me a posterior credible interval for beta12---given limited data this is going to be my approach for sure and I'm going to spend time constructing priors based on previous studies and expert knowledge. But if you want to take a frequentist approach that's fine, you can just bootstrap to obtain a confidence interval for beta12, which I'm going to be happier with than the asymptotics-based approach you take for small samples.
OK, now that I've gotten that disclaimer out of the way (apologies!), taking your paper on its own terms, here are some areas that I think could be improved.
On line 29 I'm not sure why the null hypothesis is additive; shouldn't this be a multiplicative interaction between air pollutants and nutrient intake?
Be careful in Section 2: the fact that h ~ GP with covariance kernel k means that the probability that h is in the RKHS H_k is 0! This is a surprising fact (see Wahba 1990 and Lukic and Beder 2001) and I don't think it invalidates your approach (e.g. E[h] is in the RKHS), but it's worth fixing this point.
I like the GP-LMM connection and think it needs more explanation as I am not very familiar with REML. Could I, for example, posit the following model:
y = h1 + h2 + epsilon, h1 ~ GP(0,K1) and h2 ~ GP(0,K2) and then use your approach to decide between these models:
y = h1 + epsilon
y = h2 + epsilon
y = h1 + h2 + epsilon?
I think the answer is yes, and I'm wondering if it'd be clearer to explain things this way before introducing the garrote parameter delta. 
Turning the test statistic into a model residual also helped me think about what was going on, so maybe it'd be good to move that earlier than Section 4. It also might be nice to show some training/testing curves (to illustrate bias/variance) in Section 4 to explain what's going on.
It wasn't obvious to me why the additive main effects domain H1 \osum H12 is orthogonal to the product domain H1 \otimes H2 (section 5).
For the experiments, if this were real data I could imagine trying all of the difference kernel parameterizations and choosing the one that the fits the best (based on marginal likelihood I guess) as the model for the null hypothesis. Of the different models you considered, which one fit the data best? And did this choice end up giving good type 1 and type 2 error?
A real data experiment would be nice.