Sublinear Time Orthogonal Tensor Decomposition *
A recent work (Wang et. al., NIPS 2015)  gives the fastest known algorithms for orthogonal tensor decomposition with provable guarantees. Their algorithm is based on computing sketches of the input tensor, which requires reading the entire input. We show in a number of cases one can achieve the same theoretical guarantees in sublinear time, i.e., even without reading most of the input tensor. Instead of using sketches to estimate inner products in tensor decomposition algorithms, we use importance sampling. To achieve sublinear time, we need to know the norms of tensor slices, and we show how to do this in a number of important cases. For symmetric tensors T = k i=1 λ i u ⊗p i with λ i > 0 for all i, we estimate such norms in sublinear time whenever p is even. For the important case of p = 3 and small values of k, we can also estimate such norms. For asymmetric tensors sublinear time is not possible in general, but we show if the tensor slice norms are just slightly below T F then sublinear time is again possible. One of the main strengths of our work is empirical -in a number of cases our algorithm is orders of magnitude faster than existing methods with the same accuracy.
This paper presents a randomized method for decomposing a symmetric, orthogonal tensor that is “nearly low rank”. Specifically, the tensor should expressible as a low-rank tensor + an arbitrary noise tensor with bounded norm. The fact that the tensor is symmetric and composed of orthogonal components may sound restrictive, but this is exactly the sort of tensor decomposition problem that comes up in spectral methods that solve Latent Dirichlet Allocation problems. So, it’s an interesting model to study.  The method introduced very closely follows work in a paper from NIPS 2015, “Fast and guaranteed tensor decomposition via sketching” which uses a standard tensor power method for decomposition, but speeds up each iteration by accelerating the required tensor contractions (vector dot products against the tensor) via randomized sketching techniques. This paper uses the same algorithm, but instead of using sketches based on “random projections”, which take random linear combinations of all of the components of the tensor, it uses a subsampling technique to approximate the contractions. The authors main selling point is that this approach should allow for better running times since they don’t have to touch every entry in the tensor with each iteration.  Sampling also avoids and expensive preprocessing cost incurred by the sketching algorithm.  Without knowing the prior work, it’s a bit tough to understand how the runtime of this algorithm compares. In the introduction it’s stated that the sampling algorithm is slower per iteration than the sketching methods, which are accelerated with sparse hashing and fast Fourier methods. However, runtimes are stated in terms of the sketching dimension b which is not the same for both algorithms. It can vary by orders of magnitude in the experiments. It’s stated that the algorithm has a faster runtime overall, I believe because of preprocessing costs in the sketching algorithm, but I would like to see these costs laid out so that the overall runtime comparison is clear.  Regardless, the algorithm performs well experimentally, providing a (3x-6x) speedup for small datasets and up to a 50x speedup for large synthetic data sets. This is especially impressive since the sampling algorithm is much simpler than the prior work, whose implementation seemingly requires many optimizations and detailed parameter setting.
This paper is decently written and the experimental results look promising. However, I don’t think the result is quite strong enough that I would recommend acceptance for NIPS. In general, the paper leans heavily on the work in “Fast and guaranteed tensor decomposition via sketching”. The idea to use sampling is nice, but the analysis is straightforward since the sampling scheme plugs into the prior work black-box. The authors do not prove any tighter theoretical guarantees and it’s not clear if their methods give any provable runtime improvements over the sketching methods. I would encourage the authors to clarify this aspect of their paper.    More generally, while it’s nice to have theoretical guarantees, I worry that the bound in Lemma 10 is too complex/loose to provide indication of the algorithm’s quality. In addition to a very strong requirement on the noise eps, it has an inherent n^3 dependence, a stable rank dependence, a polynomial dependence on the failure probability, and spectral gap and condition number dependencies that I don’t expect to be small. It would be great to see if the analysis of these sketching techniques can be pushed beyond the work in “Fast and guaranteed tensor decomposition via sketching” to obtain a more interpretable/convincing bound. In light of it’s demonstrated practical performance, perhaps sampling can be analyzed in a tighter way? As a more specific point, I don’t understand the meaning of r(λ)=max(λ_i/λ_j). Wouldn’t this simply equal λ_1/λ_k?  As far as presentation goes, I have a couple of comments:  I think it’s a bit confusing to claim that the algorithm runs in “sublinear time”: it seems that the final runtime in Theorem 10 has a total O(n^3) dependence since b depends on n^2. I understand that that in practice the running time *can* be sublinear because the required number of samples is much lower than predicted by the theory, but I think this should be made more clear in the abstract and introduction.   For a shortened form of the paper (like the NIPS submission) I would prefer to see the sampling analysis (currently in the supplemental material) in the main text, in lieu of the discussion of how to very efficiently generate importance samples. While I appreciate the careful work on this aspect of the algorithm (the actually sampling process is often overlooked by other papers!) ultimately it’s focusing on smallish log factors and is somewhat orthogonal to the main message of the paper. It could be relegated to an appendix.