Efficient Sampling for Learning Sparse Additive Models in High Dimensions
We consider the problem of learning sparse additive models, i.e., functions of the form:
Here S is an unknown subset of coordinate variables with |S| = k d. Assuming φ l 's to be smooth, we propose a set of points at which to sample f and an efficient randomized algorithm that recovers a uniform approximation to each unknown φ l . We provide a rigorous theoretical analysis of our scheme along with sample complexity bounds. Our algorithm utilizes recent results from compressive sensing theory along with a novel convex quadratic program for recovering robust uniform approximations to univariate functions, from point queries corrupted with arbitrary bounded noise. Lastly we theoretically analyze the impact of noise -either arbitrary but bounded, or stochastic -on the performance of our algorithm.
This paper considers the problem of estimating the components of a high-dimensional function f(x), where x is a vector of dimension d, which follows a sparse additive model, i.e. f(x) can be written as the sum of a small number of functions f_i(x_i). The authors propose sampling this function at regular points on a certain straight line and additionally on random vertices of a small hypercube surrounding these points. A convex program is then used to obtain an approximation of the gradient of f, exploiting the assumption that it is sparse. 4th order splines are employed to estimate the components of f from this approximation. The authors provide theoretical analysis showing that the method is guaranteed to detect the correct components of f in the absence of noise and also that the procedure is robust to noise.The main contribution of this paper is an innovative approach to perform robust estimation of gradients in high dimensions through randomized sampling. Although the theoretical analysis follows from standard compressed-sensing results, the idea is clever and the connection to regression with sparse additive models is intriguing. The second step of the algorithm, where the gradient estimates are used to estimate the individual components is less innovative. The authors also provide some useful analysis of the stability of the algorithm, which provides insight into the relationship between the grid size, the curvature of the components and robustness to noise (unfortunately, no numerical simulations are provided to support the theoretical conclusions).The following are the weakest aspects of the paper:-The authors do not exploit the group-sparse structure that is implicit in their model. They propose solving a large number of sparse regression problems that have the same sparsity pattern. This suggests two ways of improving the algorithm in terms of sampling complexity and robustness to noise:1. Estimate the sparsity pattern from a few samples. Then just estimate the derivative of the nonzero components by using a finite-difference approximation in their direction.2. Employ a regularizer that promotes group sparsity while estimating the gradients, e.g. an l1-l2 norm penalty.-The data, i.e. the samples of the function of f, are only used to approximate the gradients of the components f_i. They are not taken into account when the component functions f_i are constructed from the gradient estimate. It would probably make sense to ensure that the sum of the estimated components at each sample is consistent with the data up to the noise level. -The authors do not motivate sufficiently why the model would be of interest in practical applications. A concise example of an application would go a long way to clarify this.-The numerical simulations are extremely limited. In particular, no attempt is made to illustrate the limits of the method or how the method’s performance depends on the different parameters.Quality:The paper makes an interesting algorithmic contribution and is technically sound, although there are some caveats that I have mentioned above. The numerical simulations are insufficient.Clarity:The paper is written clearly and is easy to read. Originality:The idea of approximating a sparse gradient in high dimensions by sampling a function in random directions within a neighborhood is new to the best of my knowledge. However, recovering a function from its samples by penalizing the norm of its second derivative is not as innovative.Significance:This paper is significant from a theoretical point of view in that it provides a method for estimating gradients of a function in a neighbourhood by randomized sampling and convex programming. I am hesitant to state that this is very significant because the authors do not describe any relevant applications and because of the lack of numerical experiments.Other comments:The recovery of the components from their gradients must necessarily depend on the second-order behaviour of the components with respect to the chosen grid. A bound on the second derivative does appear in the error bounds, but the authors do not elaborate on its significance.After the author's rebuttal my views on the paper have not changed. Just to be clear, according to the authors' description of the model the sparsity pattern of the sparse recovery problems they solve is fixed and determined by the set S.