Robust Sparse Principal Component Regression under the High Dimensional Elliptical Model
In this paper we focus on the principal component regression and its application to high dimension non-Gaussian data. The major contributions are two folds. First, in low dimensions and under the Gaussian model, by borrowing the strength from recent development in minimax optimal principal component estimation, we first time sharply characterize the potential advantage of classical principal component regression over least square estimation. Secondly, we propose and analyze a new robust sparse principal component regression on high dimensional elliptically distributed data. The elliptical distribution is a semiparametric generalization of the Gaussian, including many well known distributions such as multivariate Gaussian, rank-deficient Gaussian, t, Cauchy, and logistic. It allows the random vector to be heavy tailed and have tail dependence. These extra flexibilities make it very suitable for modeling finance and biomedical imaging data. Under the elliptical model, we prove that our method can estimate the regression coefficients in the optimal parametric rate and therefore is a good alternative to the Gaussian based methods. Experiments on synthetic and real world data are conducted to illustrate the empirical usefulness of the proposed method.
Response to author feedback: Thank you for 
      clarifying the novelty of the robust PCA approached; it is a good idea to 
      also describe the novelty saliently in the paper. I suggest also 
      explicitly mentioning the possibility of the generalization to the case 
      with more than one PCA component, even if the full proof would not fit in 
      this paper. Simply saying in the Introduction that you consider here the 
      special case of one component, since it is already an interesting case 
      backed up with positive empirical experiments, would already help a lot. 
      Summary: The paper generalizes principal component 
      regression (PCR) by providing a robust variant of the technique based 
      on multivariate Kendall's tau in place of the covariance matrix. The 
      authors then provide a simple algorithm for learning the model and, 
      more importantly, analyze its theoretical properties, providing the rate 
      of convergence. Finally, the model is demonstrated on simulated and 
      equity data. Quality: The method presented in the paper is 
      fairly straightforward, consisting merely of estimating the population 
      Kendall's tau, computing its first eigenvector, and then performing 
      linear regression to the observations. The method is very intuitive; 
      it is easy to understand why this should be preferred to classical 
      PCR when working with data that has outliers. The use of Kendall's 
      tau in place of the covariance matrix in PCA is a nice idea and also 
      well justified based on Oja's theorem. However, it remains unclear 
      whether this should be considered as a novel contribution of the 
      paper; no citations for earlier works are given, but the authors do 
      not seem to describe it as a key contribution of the paper either. I 
      believe it has potential for wider impact than the particular 
      example considered in this paper, and one could imagine already a 
      paper studying such a robust PCA estimator as a valuable contribution. 
      To my knowledge, the closest work here would be the very recent paper 
      by Han and Liu in ICML'13. Can the authors clarify the relationship 
      with that paper? The analysis of the theoretical properties of PCR 
      is valuable. However, the whole discussion is limited to the 
      special case of simple PCR where the outcome is assumed to be 
      related only to the first principal component. Do any of the 
      results generalize to the more general case where the outcome is 
      related to the first K components? At least the authors should 
      explicitly mention that they limit the theoretical analysis to this 
      special case; now the paper never even mentions the PCR setup I would 
      consider as the standard one. The experiments seem to be conducted 
      properly and they clearly illustrate the advantage of the robust 
      variant; such a set is sufficient for a theoretical paper. I like the 
      fact that the authors show how the equity data is not normally 
      distributed, motivating the example. Clarity: The 
      paper is reasonably well written, to the degree that a theory-heavy 
      paper can be. However, the exact scope and contributions are a bit 
      vague; it is unclear whether the use of Kendall's tau for robust PCA 
      is novel, the authors do not mention they limit the analysis to a 
      special case of PCR, and some of the earlier results are listed as 
      "well known" without citations. One problem is that the proofs 
      for the perhaps main contributions of the paper, Theorems 3.2 and 3.3, 
      are left for Supplementary material, without even mentioning it in the 
      paper itself. It is understandable that writing out the full 
      proofs would take too much space here, but some sort of an outline 
      would be useful. Originality: The paper has two novelties: 
      it presents a novel robust PCR technique and it provides new 
      theoretical analysis of PCR in general. The significance of the first 
      relies fully on whether the use of Kendall's tau in PCA should be 
      considered as a novel contribution; if not, the algorithm for robust 
      PCR is trivial. The theoretical analysis provides new useful 
      results and is based on very recent works by Oja, Vu and Lei, and 
      Ravikumar et al. Significance: The robust method is 
      clearly useful for anyone applying PCR, especially in light of the 
      theoretical analysis for the convergence rate. However, the 
      significance may be limited due to the fact that both the theoretical 
      analysis and the simulation experiment rely on the simple model where 
      the output only depends on the first principal component. The 
      application to the equity data suggests the method works well in 
      practice, but is merely one example. Detailed comments: 
      - The 2nd sentence of Abstract is very complex and tries to lay 
      out all the dimensions of the paper at once. You should consider 
      splitting it into at least two sentences. - Page 3: "As a 
      well-known result..." would need a citation. In general, "well known" 
      is not good use of scientific language; you use it twice in the same 
      paragraph for results that are not obvious for most readers. - 
      Section 3.1: It is good to remind the readers about the elliptical 
      distribution, but listing all the equivalent formulations is not be 
      necessary when the rest of the analysis only uses one of them.