SPALS: Fast Alternating Least Squares via Implicit Leverage Scores Sampling
Tensor CANDECOMP/PARAFAC (CP) decomposition is a powerful but computationally challenging tool in modern data analytics. In this paper, we show ways of sampling intermediate steps of alternating minimization algorithms for computing low rank tensor CP decompositions, leading to the sparse alternating least squares (SPALS) method. Specifically, we sample the Khatri-Rao product, which arises as an intermediate object during the iterations of alternating least squares. This product captures the interactions between different tensor modes, and form the main computational bottleneck for solving many tensor related tasks. By exploiting the spectral structures of the matrix Khatri-Rao product, we provide efficient access to its statistical leverage scores. When applied to the tensor CP decomposition, our method leads to the first algorithm that runs in sublinear time per-iteration and approximates the output of deterministic alternating least squares algorithms. Empirical evaluations of this approach show significant speedups over existing randomized and deterministic routines for performing CP decomposition. On a tensor of the size 2.4m ⇥ 6.6m ⇥ 92k with over 2 billion nonzeros formed by Amazon product reviews, our routine converges in two minutes to the same error as deterministic ALS.
This submission is concerned with approximate low-rank tensor approximation. Given a, say, 3-dimensional tensor T and a target rank R, the goal is to find vectors families of vectors a_r, b_r, and c_r for r=1,...,R such that T is well approximated by sum_{r=1}^R a_r * b_r * c_r, where here * denotes outer product. Henceforth A is the matrix with a_r as columns, etc. Error in the approximation is measured using the Frobenius norm. The submission focuses on one algorithm to solve this problem, namely alternating least squares (ALS), and discusses how to speed it up using leverage score sampling. Specifically, ALS in this context fixes 2 out of 3 of A, B, C, then tries to minimize the Frobenius norm error by optimizing the third. It then loops over these 3 families as the variable to optimize. The main observation of the submission is that once 2/3 of these matrices are fixed, the optimization problem is equivalent to a traditional least squares problem in which the matrix X is n^2 x n, and now standard results in the randomized linear algebra literature thus imply that it suffices to sample only O(n log n) rows of X via nonuniform sampling of the rows of X via its statistical leverage scores. Unfortunately computing the statistical leverage scores of X is expensive, since it has n^2 rows. The main observation of this paper is that for ALS, X has special structure (it is the "Khatri-Rao product" of the two matrices that were fixed), and a lemma is then proven in the submission showing that one can upper bound the leverage scores of X by simple functions of the leverage scores of the two matrices that were fixed. This overall leads to a faster algorithm for computing upper bounds on the leverage scores of X, which then translates into a faster ALS variant at the end of the day.
As the "LS" of ALS suggests, each iteration of ALS for tensor decomposition amounts to solving a least squares regression problem. The main contribution of this submission is then to observe that good upper bounds on the leverage scores of the underlying matrix can be quickly approximated due to special structure of the matrix, namely Theorem 3.2 of the submission. This is the only, albeit important, novel observation of this paper. Once Theorem 3.2 is obtained, filling in the other details is standard. From this one observation, they are able to compare quite favorably with [37] (see Figure (a) on page 8).  The plus side is clear: the ability to compete well with the state-of-the-art based on quite a simple observation. The two main downsides I see are: (1) they seem to be more comparable with [37] in empirical performance as opposed to hands down beating it, and (2) in figure (b) I didn't see any explanation of why again [37] wasn't compared with.  I was also confused as to why SPALS(alpha) figure numbers weren't monotonic with alpha in figure (a) on page 8.  Minor comments: * Some typos: "nearly optimality" --> "near optimality"; "while approximates the" --> "while approximating the"; "for large tensor" --> "for large tensors"; "each rank-1 components are" --> "each rank-1 component is"; "It is a challenging tasks" --> "It is a challenging task"; --> "one of the most powerful tool" --> "one of the most powerful tools"; "we provide the efficient" --> "we provide an efficient"; "In the remaining of this section" --> "In the remainder of this section"; "toolset for estimating the statistical leverage score" --> "toolset for estimating the statistical leverage scores"; "the statistical leverage scores of the i-th row" --> "the statistical leverage score of the i-th row"; "score of certain row" --> "score of a certain row"; "first inequality is because of" --> "first inequality is because"; "equals to $R$" --> "equals $R$"; "an rank-1" --> "a rank-1"; "In the remaining of this section" --> "In the remainder of this section"; "separate the calculation to two parts" --> "separate the calculation into two parts"; "evaluating former expression" --> "evaluating the former expression"; "second term is spare" --> "second term is sparse"; "by the leverage score of the design" --> "by the leverage scores of the design"; "not directly utilize" --> "not directly utilizing"; "requires an one-time" --> "requires a one-time"; "routines fairly recent" --> "routines was fairly recent"; "moves the non-zeros around" --> "moves the non-zeroes around"; "considered as the data" --> "considered as data"; "piecemeal invocation" --> "piecemeal invocations"; "each rank-1 components" --> "each rank-1 component" * On page 2, line 80 "(i,j) also represents the index i+Ij between 1 and IJ"; I found this confusing. What does it mean? * On page 4, lines 128 and 129: "Its optimality in solving ... of linear regression." It cannot really be explained by this; it's not so trivial. The upper bound requires using some matrix concentration, like matrix Bernstein or matrix Chernoff, or the non-commutative Khintchine inequality. * On page 4, line 133: "O(r log n)"; why did n enter the picture? I don't know any results of this form where n enters the picture.  * I found Figure (a) on page 8 to be confusing. First, how is "error" measured? Also, why is the SPALS(alpha) error not monotonic as a function of alpha? What are the units on the time and error? * On page 8, does n = 1000 mean the tensor is 1000 x 1000 x 1000? (All 3 dimensions are n, right?)