Group Additive Structure Identification for Kernel Nonparametric Regression
The additive model is one of the most popularly used models for high dimensional nonparametric regression analysis. However, its main drawback is that it neglects possible interactions between predictor variables. In this paper, we reexamine the group additive model proposed in the literature, and rigorously define the intrinsic group additive structure for the relationship between the response variable Y and the predictor vector X, and further develop an effective structure-penalized kernel method for simultaneous identification of the intrinsic group additive structure and nonparametric function estimation. The method utilizes a novel complexity measure we derive for group additive structures. We show that the proposed method is consistent in identifying the intrinsic group additive structure. Simulation study and real data applications demonstrate the effectiveness of the proposed method as a general tool for high dimensional nonparametric regression.
Note: Since the supplement appears to include the main paper, I simply reviewed that, and all line numbers below correspond to the supplement.
Summary:
This studies group-additive nonparametric regression models, in which, for some partitioning of the predictor variables, the regression function is additive between groups of variables; this model interpolates between the fully nonparametric model, which is difficult to fit, and the additive model, which is sometimes too restrictive. Specifically, the paper studies the problem where the group structure is not known in advance and must be learned from the data. To do this, the paper proposes a novel penalty function, based on the covering numbers of RKHS balls, which is then added to the kernel ridge regression objective. This results in an objective that can be optimized over both the group structure (which, together with the kernel determines a function space via direct sum of RKHSs over each group of variables) and the regression estimate within each group. Two algorithms are presented for approximately solving this compound optimization problem, and then theoretical results are presented showing (a) the rate at which the empirical risk of the estimate approaches the true risk of the true optimum, and (b) consistency of the group structure estimate, in that the probability it matches the true group structure approaches 1 as n -> infinity. Finally, experimental results are presented on both synthetic and real data.
Main Comments:
The key innovation of the paper appears to be recognizing that the complexity of a particular group structure can be quantified in terms of covering numbers of the direct sum space. The paper is comprehensive, including a well-motivated and novel method, and reasonably solid theoretical and empirical results. I'm not too familiar with other approaches to fitting models between the additive and nonparametric models, but, assuming the discussion in Lines 31-50 is fairly, complete, this paper seems like a potentially significant advance. As noted in the Discussion section, the main issue with the method appears to be difficulty solving the optimization problem over group structure when the number of variables in large. The paper is also fairly clearly written, aside from a lot of typos.
Minor Comments/Questions:
Just curious: is there any simple characterization of "interaction" between two variables that doesn't rely on writing the whole model in Equation (1)?
Line 169: I don't quite understand the use of almost surely here. Is this meant as n -> infinity?
Equation (3) is missing a summation (over i).
Line 205: Perhaps "translation invariant kernel" is a more common term for this than "convolutional kernel"
Typos:
Line 145: "on RHS" should be "on the RHS"
Line 152: "not only can the bias of \hat f_{\lambda,G} reduces" should be "not only can the bias of \hat f_{\lambda,G} reduce"
Line 160: "G^* exists and unique" should be "G^* exists and is unique"
Line 247: "turning parameters" should be "tuning parameters"
Algorithm 2: Line 1: "State with" should be "start with"
Line 251: "by compare" should be "by comparing".