Fast Algorithms for Robust PCA via Gradient Descent
We consider the problem of Robust PCA in the fully and partially observed settings. Without corruptions, this is the well-known matrix completion problem. From a statistical standpoint this problem has been recently well-studied, and conditions on when recovery is possible (how many observations do we need, how many corruptions can we tolerate) via polynomial-time algorithms is by now understood. This paper presents and analyzes a non-convex optimization approach that greatly reduces the computational complexity of the above problems, compared to the best available algorithms. In particular, in the fully observed case, with r denoting rank and d dimension, we reduce the complexity from O(r 2 d 2 log(1/ε)) to O(rd 2 log(1/ε)) -a big savings when the rank is big. For the partially observed case, we show the complexity of our algorithm is no more than O(r 4 d log d log(1/ε)). Not only is this the best-known run-time for a provable algorithm under partial observation, but in the setting where r is small compared to d, it also allows for near-linear-in-d run-time that can be exploited in the fully-observed case as well, by simply running our algorithm on a subset of the observations.
This paper presents an efficient algorithm for the problem of robust PCA. The key idea is estimating a good initial solution followed by gradient descent for improvement. Compared to a recent (non-convex) robust PCA solver, the computational complexity is reduced by a factor of "r", where "r" denotes the rank of the true matrix. The algorithm is straightforward, easy to follow. The main technical contribution is the linear convergence guarantee which appears novel.
This is an interesting work that presents the perhaps first theoretical guarantee for a widely used optimization technique of the problem of robust PCA. I think it has potential and should be somewhere between poster and oral. During the rebuttal period, I would suggest the authors to address the following minor concerns.  - It appears to me that the algorithms (Algorithm 1 and Algorithm 2) is easy to follow. That been said, it is not clear how one is able to know the true rank "r" and the corruption fraction "alpha". The projection step (Step 7 and Step 8) requires a knowledge of the incoherence parameter. How to estimate it? If one only knows some upper bound on the rank and corruption fraction (which should be the practical case), is it possible to show that all the theoretical results still hold?  - Theorem 1 states that the initial deviation for "(U_0, V_0)" is upper bounded by a quantity which is proportional to "sqrt(r/kappa)". This means when "r" is large, the initial estimate is not good enough. Given this, why does the algorithm still succeed by projecting the iterates onto the space of "mathcal{U}" and "mathcal{V}" which are determined by "U_0" and "V_0", respectively? Please give more intuition on it.  - The sample complexity is not optimal here. The authors claim that it can be improved by utilizing the output of [21]. For the sake of concreteness, the authors should frame such an argument more rigorously, e.g., "if we apply the AltProj[21] for initialization, our Theorem 1 can be strengthened to d(U_0, V_0; U^*, V^*) < ... In this way, we obtain an improved robustness result ..." Otherwise, it is tricky to check the correctness of Remark 2.  - When analyzing the computational complexity and robustness, the authors presume that "kappa" is a constant and compare the theoretical result to state-of-the-art solvers. This is somewhat unfair since "kappa" might scale with the dimension. A good practice is carefully discuss when the presented algorithm dominates previous solvers and when it does not, e.g., "when kappa = O(1), we observe that our algorithm is an order of magnitude faster ... when kappa = O(d), we find that ...". It would be better to provide several examples where "kappa"  behaves like a constant.   --------------------------------- Updates:  The author feedback is convincing. It is a nice work to me.  A minor comment: It seems that another parallel work [A] also considers online alternating minimization. It would be good if the authors could comment on the difference.  [A] Provable Efficient Online Matrix Completion via Non-convex Stochastic Gradient Descent, arXiv:1605.08370