Fast and Guaranteed Tensor Decomposition via Sketching
Tensor CANDECOMP/PARAFAC (CP) decomposition has wide applications in statistical learning of latent variable models and in data mining. In this paper, we propose fast and randomized tensor CP decomposition algorithms based on sketching. We build on the idea of count sketches, but introduce many novel ideas which are unique to tensors. We develop novel methods for randomized computation of tensor contractions via FFTs, without explicitly forming the tensors. Such tensor contractions are encountered in decomposition methods such as tensor power iterations and alternating least squares. We also design novel colliding hashes for symmetric tensors to further save time in computing the sketches. We then combine these sketching ideas with existing whitening and tensor power iterative techniques to obtain the fastest algorithm on both sparse and dense tensors. The quality of approximation under our method does not depend on properties such as sparsity, uniformity of elements, etc. We apply the method for topic modeling and obtain competitive results.
# Summary The paper applies existing tensor sketching ideas to orthogonal tensor decomposition, leading to an algorithm with running time O(n^2 log(n)) (vs. O(n^3)), when the eigenvalue spectrum decays superlinearly. The authors propose a new hashing method for symmetric tensors, and validate their computational speedups on topic models.# Comments * The idea of storing a low-dimensional sketched representation is exciting, however the results presented in Theorem 2 only hold when the spectrum of the tensor (the eigenvalues) decay super-linearly, as in the synthetic experiments (e.g. $\lambda_i = 1/i$). In the case where the spectrum is flat, i.e. $\lambda_i$ are approximately the same, the size of the sketch required is $O(n^3)$, according to Theorem 2. Is this an artifact of the analysis or a limitation of the approach? * In the experiments, and in table 3, the sketch building time is ignored -- what exactly comprises of sketch building time? It seems like this is a crucial component to the method. * How does the method scale with the magnitude of error ($\sigma$)? * Is the sketching of a rank-1 tensor (3.2.1) a contribution of the current paper? It appears to be presented in [22; Algorithm 1]. * How does the symmetric tensor sketch proposed in 3.4 differ from the idea of using the complex ring to count higher moments, e.g. Ganguly, Sumit. "Estimating frequency moments of data streams using random linear combinations." Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques. Springer Berlin Heidelberg, 2004. 369-380.# Minor commments * The notation in the paper is hard to follow and very densely packed. In some places, the notation is not clearly defined, e.g. 3.1., [n] was not defined, and $p$ is not described. * In Algorithm 2, the footnote 4 is easily confused with an exponent. Also, it is not clear whether line 7 has a typo: "$[v^{(m)}]_i \gets \ksi_1(i) ..$" <- is it always $\ksi_1$? * What is meant by time constraints when describing the synthetic experiments (margin line #392).