Early Stopping for Nonparametric Testing
Early stopping of iterative algorithms is an algorithmic regularization method to avoid over-fitting in estimation and classification. In this paper, we show that early stopping can also be applied to obtain the minimax optimal testing in a general non-parametric setup. Specifically, a Wald-type test statistic is obtained based on an iterated estimate produced by functional gradient descent algorithms in a reproducing kernel Hilbert space. A notable contribution is to establish a "sharp" stopping rule: when the number of iterations achieves an optimal order, testing optimality is achievable; otherwise, testing optimality becomes impossible. As a by-product, a similar sharpness result is also derived for minimax optimal estimation under early stopping. All obtained results hold for various kernel classes, including Sobolev smoothness classes and Gaussian kernel classes.
Summary: This is a very interesting paper that provides an alternative approach to nonparametric testing. Instead of imposing a penalty to encourage bias-variance tradeoff, the paper proposed to do an “early stopping” to achieve that. The reason why this approach would work is that when applying a gradient ascent/descent approach, even the optimal solution will overfit the data, on the way to the optimal, there is a sweet spot where the variability has been removed a lot while the bias is still not too large (have not yet overfit the data).  Overall, this is a nice and well-written paper and its idea is worth spreading in the community of machine learning and statistics.    Detail comments: 1. In Theorem 3.1 and soon after it, there is a rule for testing H0 using the asymptotic distribution.  However, if I understand it correctly, this rule will be asymptotically valid when both t and n goes to infinity.  Will the stopping rule T^* diverges when n goes to infinity?  Otherwise it may not really control the asymptotic type-1 error.   2. Although the power analysis has been done for the rejection rule, the type-1 error at t=T^* was not clear. In particular, the stopping rule T^* also depends on the data. So Theorem 3.1 may not be enough to guard the type-1 error. Is there any theoretical results about the control of type-1 error?  There might be two possible ways to do that. One is to derive the asymptotic distribution at t=T^* when n goes to infinity.  The other is via the optional stopping theorem.   3. Will this method be applicable to estimation problem? I guess the main challenge is to find the right time to stop the gradient ascent so the stopping has to be re-defined. Maybe the idea of Lepski’s adaptive rule could be useful in this case.