Online PCA for Contaminated Data
We consider the online Principal Component Analysis (PCA) where contaminated samples (containing outliers) are revealed sequentially to the Principal Components (PCs) estimator. Due to their sensitiveness to outliers, previous online PCA algorithms fail in this case and their results can be arbitrarily skewed by the outliers. Here we propose the online robust PCA algorithm, which is able to improve the PCs estimation upon an initial one steadily, even when faced with a constant fraction of outliers. We show that the final result of the proposed online RPCA has an acceptable degradation from the optimum. Actually, under mild conditions, online RPCA achieves the maximal robustness with a 50% breakdown point. Moreover, online RPCA is shown to be efficient for both storage and computation, since it need not re-explore the previous samples as in traditional robust PCA algorithms. This endows online RPCA with scalability for large scale data.
This paper considers the theoretical properties of 
      online updating of a "robust" PCA. Unlike traditional RPCA the proposed 
      method does not learn the usual decomposition of the data into low-rank 
      and sparse components but instead outliers are handled by accepting 
      observations into the estimate of the covariance matrix proportional to 
      their variance along the previously estimated PCA directions and scaling 
      the accepted observations by this probability. The proposed 
      approach seems novel and certainly looks to be simpler than other 
      approaches solving the same problem which are referred to in the paper. 
      However it would have been nice to see a comparison of these methods. 
      The technical contribution of the paper seems sound and 
      illuminating however the important question remains: how likely is it that 
      the initial condition on H(w^0) is satisfied. This also seems to limit the 
      usefulness of the method in real applications since in the experiments the 
      authors initialise with PCA computed with the known outliers removed -- 
      perhaps RPCA or similar could be used to get the initial estimate? 
      It also seems like in high dimensions, there might be issues due 
      to high dimensionality/low sample size when computing the initial 
      estimates. There are a few other questions I am left with: 
      What happens when the concept changes i.e. when the underlying data 
      generating process changes? I have also noticed some possibly 
      minor mistakes/typos. In section 3.1 some random variable \mu is 
      mentioned but it is never defined and seemingly not mentioned again. Is 
      this a typo? The sentence introducing \lambda is quite strange, it 
      implies that \lambda has already been defined but in fact it has not. 
      In several places "affect" is mixed up with "effect". 
      Captions in Figs 2 and 3 are wrong. They should be number of 
      batches. It would have been instructive to compare the results 
      with batch mode RPCA and also report on timings to get an idea if there is 
      a significant speedup.