Provable Tensor Factorization with Missing Data
We study the problem of low-rank tensor factorization in the presence of missing data. We ask the following question: how many sampled entries do we need, to efficiently and exactly reconstruct a tensor with a low-rank orthogonal decomposition? We propose a novel alternating minimization based method which iteratively refines estimates of the singular vectors. We show that under certain standard assumptions, our method can recover a three-mode n × n × n dimensional rank-r tensor exactly from O(n 3/2 r 5 log 4 n) randomly sampled entries. In the process of proving this result, we solve two challenging sub-problems for tensors with missing data. First, in analyzing the initialization step, we prove a generalization of a celebrated result by Szemerédie et al. on the spectrum of random graphs. We show that this initialization step alone is sufficient to achieve the root mean squared error on the parameters bounded by C(r 2 n 3/2 (log n) 4 /|Ω|) from |Ω| observed entries for some constant C independent of n and r. Next, we prove global convergence of alternating minimization with this good initialization. Simulations suggest that the dependence of the sample size on the dimensionality n is indeed tight.
This paper proposes an algorithm for the exact recovery of low-rank tensors from randomly chosen O(r^5n^{3/2}log^4(n)) entries (in the case of rank-r symmetric n x n x n tensor).The proposed algorithm uses the robust tensor power method of Anandkumar et al. [1] to obtain an inital solution for the least squares iteration, which is a popular approach for tensor factorization. The proof relies on a rather strong notion of incoherence of the factor matrix and the proposed algorithm relies on the knowledge of the correct rank and the incoherence parameter.The theoretically predicted scaling is confirmed in a synthetic experiment except that the authors heuristically assume a milder dependency with respect to the rank r (r^{1/2} instead of r^5).The strength of the paper is that as far as I know, this is the first paper that presents a tensor completion algorithm with a O(n^{3/2}) guarantee from randomly chosen entries, though the authors have missed two related studies: Krishnamurthy & Singh (NIPS 2013) have shown a O(n) sample complexity when the samples are adaptively chosen; Tomioka et al. (NIPS 2011) have analyzed the random design setting and have shown a O(n^2) sample complexity.Another interesting point is that it uses the robust tensor power method as an initalizer for a more conventional iterative algorithm, which have shown to be an effective strategy by other studies (Chaganty & Liang, ICML 2013).On the negative side, the incoherence assumption employed by the authors seems rather strong. It basically ensures that all the entries of the true tensor T have the magnitude of O(n^{-3/2}). Compared to the fact that it is possible to show a tight sample complexity when the entries have magnitude O(1) in matrix completion (Foygel & Srebro, COLT2011), this is rather disappointing.The algorithm also assumes the knowledge of the correct rank r and the incoherence parameter mu. It was not clear to me what happens if these parameters were mis-specified.Minor comments:- sigma_max and sigima_min whould be defined when they first appear (in page 3).- I am not sure if the claim "the above theorem implies that among n^2 fibers of the form ... it is sufficient to have only O(n^{3/2}) fibers with any samples" (page 4) is true. At least it is not implied by the theorem because the theorem assumes uniform sampling. For example if only the first entries for all 3n^2 fibers were sampled, would this be enough to recover any incoherent low-rank tensor? This is not true for r > 1 for matrix completion because removing any observed entry would make the graph disconnected (see Kiraly and Tomioka, ICML 2012).- The claim that the degrees of freedom of a symmetric 3rd order tensor is rn-r^2 (page 4) needs a proof or a reference.