A statistical model for tensor PCA
We consider the Principal Component Analysis problem for large tensors of arbitrary order k under a single-spike (or rank-one plus noise) model. On the one hand, we use information theory, and recent results in probability theory, to establish necessary and sufficient conditions under which the principal component can be estimated using unbounded computational resources. It turns out that this is possible as soon as the signal-to-noise ratio β becomes larger than C √ k log k (and in particular β can remain bounded as the problem dimensions increase). On the other hand, we analyze several polynomial-time estimation algorithms, based on tensor unfolding, power iteration and message passing ideas from graphical models. We show that, unless the signal-to-noise ratio diverges in the system dimensions, none of these approaches succeeds. This is possibly related to a fundamental limitation of computationally tractable estimators for this problem. We discuss various initializations for tensor power iteration, and show that a tractable initialization based on the spectrum of the unfolded tensor outperforms significantly baseline methods, statistically and computationally. Finally, we consider the case in which additional side information is available about the unknown signal. We characterize the amount of side information that allows the iterative algorithms to converge to a good estimate.
Description:The paper presents a theoretical analysis of exact and approximate MLestimation of a simple form of tensor factorization. Namely, we aregiven a tensor X (of rank k and dimension n) which equals the k-foldtensor product of some vector v_0 with itself, plus a noise tensorwith components being iid Gaussian. The task is to estimate v_0 fromX. The task leads to a non-convex optimization problem overv_0.Assuming non-zero noise, the task is NP hard. The paper assumesseveral known algorithms to solve the task approximately:matricization (reshaping X to a matrix and solving matrixfactorization), power iterations, message passing, and message passinginitialized by a guiding information which is the true value of v_0plus noise.The following is proved:1) The difference between the exact ML estimate (requiring exponentialtime) of v_0 and the true v_0 is small with high probability if SNR isgreater than a threshold, which is a polynomial function of k. Thismeans, the exact ML estimation is succesfull for a reasonable SNR thatscales well with tensor rank.2) All approximate ML estimates with no guiding information (i.e.,matricization, power iterations, and unguided message passing) aresucessfull only for SNR larger than an exponential function of k. Thatis, for larger k these methods can yield very bad estimates.3) The guiding information can make approximate message passingsucessfull, provided that it the SNR of this guiding information isreasonably low.These results are demonstrated on simple experiments on synthetic dataand k=3.Comments:The achieved results are valuable and non-trivial and I recommend toaccept the paper. However, I do have a number of objections.My most important objection: The paper presents a *statistical* analysis of exact and approximate ML estimation of tensor factorization. However, it would be helpful tofirst summarize known results on hardness of the ML estimation problemfrom computational complexity point of view, without anystatistics. How hard is to *approximate* the Tensor PCAproblem (on lines 59-60)? You only say that it is NP-hard, but is it in APX, does it have a PTAS? How do these results relate to your statistical analysis?The authors informally conjecture (see abstract) that the founddichotomy between the exact (intractable) and approximate (tractable)ML estimation can be valid in general: all tractable algorithms for MLestimation are inevitably unsuccesfull. It would be nice to see morearguments for this conjecture. E.g., do you believe that all existing tractable approximate algorithms fall more or less to one of these classes?In Theorem 3, you assume that the result of the matricizationalgorithm is the top singular vector w. But this is not really thedesired result because w has yet to be (approximately) decomposed tothe (\lceil k/2\rceil)-fold tensor product of vector v_0 withitself. Can this affect your result?In experiments, you may wish to compute also the exact ML estimate,which I believe is tractable for fixed k=3. This would empiricallysupport the results from Section 2.In experiments, considering only k=3 may hide some important insights (how does the accuracy scale with k?).Clarity of the paper should be improved. The authors do not help the reader to follow this mathematically very difficult paper. Some parts were apparentlywritten in haste. There is a number of typos, e.g.:Theorem 1: C_1 in (7) should be C_2Lemma 2: Model modelReference [18] is incomplete.Many English typos in Section 5 and in the supplement.Finally, let me admit that the topic is not exactly my expertise andtherefore verifying the proofs in detail is beyond my abilities.  Themain paper is only a summary of results and all the proofs are in thesupplement. This material is mathematically very heavy and I believeonly real specialists can understand it in detail.