Sketching Structured Matrices for Faster Nonlinear Regression
Motivated by the desire to extend fast randomized techniques to nonlinear l p regression, we consider a class of structured regression problems. These problems involve Vandermonde matrices which arise naturally in various statistical modeling settings, including classical polynomial fitting problems, additive models and approximations to recently developed randomized techniques for scalable kernel methods. We show that this structure can be exploited to further accelerate the solution of the regression problem, achieving running times that are faster than "input sparsity". We present empirical results confirming both the practical value of our modeling framework, as well as speedup benefits of randomized regression.
This paper proposes efficient randomization-based 
      algorithms for solving two types of regression problems -- corresponding 
      to linear least squares and least absolute deviation fitting over convex 
      sets -- in settings where the design matrix has a Vandermonde structure. 
      To the reviewer's best knowledge, incorporating this type of structure 
      into such randomized algorithmic approaches to linear algebra problems is 
      novel. Such methods are timely and relevant, given many recent 
      applications focused on processing and inference in "big data" regimes. 
      The paper is overall well-written and could, in the opinion of the 
      reviewer, comprise a solid contribution to the NIPS community, subject to 
      some minor revisions, described in some detail here. Primary 
      concerns: 1) The authors appear unaware of some connections 
      between the subspace embedding they describe (claimed introduced in [6]) 
      and existing efforts in the sketching literature. Specifically, the matrix 
      structure described in section 3.1.2 here was previously proposed as a 
      counting sketch in (Charikar, Moses, Kevin Chen, and Martin Farach-Colton. 
      "Finding frequent items in data streams." Automata, Languages and 
      Programming. Springer Berlin Heidelberg, 2002. 693-703.); see also, e.g., 
      the recent survey article (Gilbert, Anna, and Piotr Indyk. "Sparse 
      recovery using sparse matrices." Proceedings of the IEEE 98.6 (2010): 
      937-947.). 2) Further, there are implicit connections between the 
      embedding results on page 5 and the compressive sensing literature. 
      Namely, ensuring that the l_2 norm of vectors Ax remain preserved is 
      related under action by a linear operator (here, $\Pi$) is related to the 
      well-known restricted isometry property, albeit in this case "restricted" 
      to a single subspace of dimension equal to the number of columns in A; 
      see, for example, (Candès, Emmanuel J. "The restricted isometry property 
      and its implications for compressed sensing." Comptes Rendus Mathematique 
      346.9 (2008): 589-592.), as well as (Baraniuk, Richard, et al. "A simple 
      proof of the restricted isometry property for random matrices." 
      Constructive Approximation 28.3 (2008): 253-263.) Likewise, the l_1 
      preservation relation on page 5 here is related to the l_1-RIP condition 
      studied in (Berinde, Radu, et al. "Combining geometry and combinatorics: A 
      unified approach to sparse signal recovery." Communication, Control, and 
      Computing, 2008 46th Annual Allerton Conference on. IEEE, 2008.). 
      3) I found the experimental section somewhat unclear. 
      Specifically, it was not obvious to me specifically what the 
      classification approach was here. It would be helpful if the authors could 
      explain this in some more detail. Minor points: 1) In 
      several places, including Theorem 6 and Algorithm 2, the authors describe 
      a matrix of Gaussians. Is there any reason to prefer, e.g., unit variance 
      here? 2) On page 5, the authors state "The best known value of 
      $\kappa$, independent of $d$, is $\kappa = O(d^2 \log^2d)." How is this 
      independent of d? 3) On page 6, "Let L = (i such that a_i \neq 0 
      and ), and ..." typo? 4) Table 1 in the "CENSUS" data set, the 
      time is given as 1.4% sec One Final General Suggestion It 
      may be helpful for the reader to better appreciate the contributions here, 
      if the authors listed in each step of the proposed algorithm the 
      corresponding computational complexity. This would be especially helpful 
      where, e.g., the procedure requires QR decompositions -- this step gave 
      the reviewer pause, at first viewing.