Exploiting Numerical Sparsity for Efficient Learning : Faster Eigenvector Computation and Regression
In this paper, we obtain improved running times for regression and top eigenvector computation for numerically sparse matrices. Given a data matrix A ∈ R n×d where every row a ∈ R d has a 2 2 ≤ L and numerical sparsity at most s, i.e. a 2 1 / a 2 2 ≤ s, we provide faster algorithms for these problems in many parameter settings.
For top eigenvector computation, we obtain a running time ofÕ(nd + r(s + √ rs)/gap 2 ) where gap > 0 is the relative gap between the top two eigenvectors of A A and r is the stable rank of A. This running time improves upon the previous best unaccelerated running time of O(nd + rd/gap 2 ) as r ≤ d and s ≤ d.
For regression, we obtain a running time ofÕ(nd + (nL/µ) snL/µ) where µ > 0 is the smallest eigenvalue of A A. This running time improves upon the previous best unaccelerated running time ofÕ(nd + nLd/µ). This result expands the regimes where regression can be solved in nearly linear time from when L/µ =Õ(1) to when L/µ =Õ(d 2/3 /(sn) 1/3 ).
Furthermore, we obtain similar improvements even when row norms and numerical sparsities are non-uniform and we show how to achieve even faster running times by accelerating using approximate proximal point [9] / catalyst [15]. Our running times depend only on the size of the input and natural numerical measures of the matrix, i.e. eigenvalues and p norms, making progress on a key open problem regarding optimal running times for efficient large-scale learning.
** Summary ** The paper studied two important problems in machine learning and numerical linear algebra: top eigenvector computation and l_2 regression. The authors introduce ``numerical sparsity’’ as a new parameter to measure the efficiency of algorithms for solving these problems. For a vector x, its numerical sparsity is defined to be |x|_1^2 / |x|_2^2. This quantity is a ``softer’’ version of sparsity and is never larger than the l_0 sparsity of x. The main new insight of this paper is that, it is possible to achieve better running time if we measure the efficiency of algorithms using this new parameter. To accomplish this goal, instead of devising completely new convex optimization methods, the authors develop new stochastic gradient oracle for these problems by importance sampling the matrix A (Section 3), whose variance bound depends on the numerical sparsity of rows of A, and then directly apply existing stochastic first order methods to solve l_2 regression and top eigenvector computation (Section 4).   The main new technical result in this paper is an unbiased stochastic oracle to estimate A^T Ax for a matrix A and a vector x, which can serve as a stochastic gradient oracle when solving l_2 regression and top eigenvector computation. The variance of the oracle depends on the numerical sparsity of rows of A. I found this idea interesting and natural: consider a sparse vector a, if we slightly perturb a then its l_0 sparsity would be large. However, it is still possible to estimate such vector a accurately by (i) finding out heavy coordinates of a and (ii) importance sampling coordinates according to their magnitudes. The authors develop new stochastic oracle to estimate A^T Ax by formalizing these intuitions in Section 3. The main difficulty here is to relate the variance to the function error, and the authors achieve this goal by exploiting the numerical sparsity of rows of A. I found this sampling procedure interesting and may have applications in other machine learning / numerical linear algebra tasks.   Overall, this is a good paper and I would recommend acceptance.   ** Minor Comments ** Line 39: be achieved -> to be achieved Line 72: The definition of \lambda_min should be \lambda_d(A^T A) Line 79: (m/2) (a_i^T x - b_i) -> (m/2) (a_i^T x - b_i)^2 Line 212: In Algorithm 4, remove M in the denominator from the definition of M Line 378: how can we -> how we can Line 379: in smooth -> is smooth Line 380: remove the space after ``problems’’ Line 420: RHS of the equation should be (a^T x)^2 Line 429: The second inequality seems redundant Line 518: It seems you are assuming \kappa \ge d instead of \kappa \le d^2 in Lemma 11, since you want \sqrt{\kappa} \ge \sqrt{s_i}. This will also affect some related parts of the paper.