The Fast Convergence of Incremental PCA
We consider a situation in which we see samples X n ∈ R d drawn i.i.d. from some distribution with mean zero and unknown covariance A. We wish to compute the top eigenvector of A in an incremental fashion -with an algorithm that maintains an estimate of the top eigenvector in O(d) space, and incrementally adjusts the estimate with each new data point that arrives. Two classical such schemes are due to Krasulina (1969) and Oja (1983). We give finite-sample convergence rates for both.
This paper theoretically analyse the Oja’s algorithm 
      for estimating the first principal component. Convergence properties and 
      relation to stochastic gradient descent have been discussed. Overall, this 
      paper is not well written and should not be accepted in the current form. 
      Detail comments are: 1. All theoretical analyses are based 
      on the potential functions defined in line 109-111. There is no citation 
      for this potential function and it is not explained why this function is 
      defined in this form. Using the norm of the difference between v_n and the 
      optimal first principal component is a more standard way to analyse the 
      convergence rate. The author should explain why it is not adopted in this 
      paper. 2. The assumption of the example in section 3 is too 
      strong. It is inappropriate to assume all data points are drawn from 
      eigenvectors of the covariance matrix. It is only a special case which 
      cannot be used as a fact to analyse the initialization situation and the 
      further theoretical proofs in Section 4 and Section 5. 3. In line 
      153-155, “if v_n is orthogonal to some e_i, it will remain so forever” is 
      not true, because x_n^Tv_{n-1} is not always 0 and v_n is not always 
      equals to v_{n-1}. x_n^Tv_{n-1} = 0 only when x_n is e_i. However, x_n 
      could be e_j, where j\neq i, according to the construction of the data 
      points. 4. The submitted version is not compiled properly which 
      has many question marks when citing Appendix. The supplementary file has 
      the right citation. However, the correctness of proofs is still hard 
      to be verified because of the extremely poor organization of the 
      structure. For example: The proof of Theorem 4.1 ->A1->Theorem 
      5.2->E.1->C.1->F.1-F.3, where “->” means “refers to”. It 
      is impossible to follow unless using a clear structure. 5. This 
      paper cites too many arxiv papers. Most of these papers have been 
      published on peer reviewed journals or conferences. Please cite the peer 
      reviewed version. 6. There are many grammar mistakes which make 
      the paper difficult to read. For example, the section paragraph in 
      Introduction is not readable.