Sparse Random Features Algorithm as Coordinate Descent in Hilbert Space
In this paper, we propose a Sparse Random Features algorithm, which learns a sparse non-linear predictor by minimizing an ℓ 1 -regularized objective function over the Hilbert Space induced from a kernel function. By interpreting the algorithm as Randomized Coordinate Descent in an infinite-dimensional space, we show the proposed approach converges to a solution within ϵ-precision of that using an exact kernel method, by drawing O(1/ϵ) random features, in contrast to the O(1/ϵ 2 ) convergence achieved by current Monte-Carlo analyses of Random Features. In our experiments, the Sparse Random Feature algorithm obtains a sparse solution that requires less memory and prediction time, while maintaining comparable performance on regression and classification tasks. Moreover, as an approximate solver for the infinite-dimensional ℓ 1 -regularized problem, the randomized approach also enjoys better convergence guarantees than a Boosting approach in the setting where the greedy Boosting step cannot be performed exactly.
The paper presents a random feature based approximation algorithm for solving the l1 regularized problem in a (possibly infinite dimensinal) Hilbert space. 		    The paper is well written and easy to read. Theorem 2 and its corollaries are interesting and form the key technical contribution. 		    Solving l1 regularized problem in Hilbert space was considered ealier (for eg. [1*], which should perhaps be cited). However the proposed random feature algorithm and more importantly, its theoretical analysis are new and non-trivial. 		    Comments:		    1. Prior work on solving l1 regularized problem in Hilbert spaces perhaps need to be summarized and cited. For e.g. [1*]. Also, alternatives like [2*] may be appropriately discussed.		    2. The empirical section may be strengthened by comparing with [1*].		    3. The current choices of \lamda seem arbitrary. Why not cross-validate for accuracy for all algorithms and report support-vectors/sparsity too? I think the current choice makes the values in the plots un-comparable. Or alternatively, provided plots with varying \lambda.		    4. Some discussion on the results is needed, for e.g, performance in case of regression seems to be better than classification. why? etc.	    [1*]. S. Rosset et.al. l1 regularization ininfinite dimensional feature spaces. COLT-2007.	    [2*]. G. Song et.al. Reproducing kernel banach spaces with the l1 norm. Journal of Applied and Computational Harmonic Analysis.