Generalizing Analytic Shrinkage for Arbitrary Covariance Structures
Analytic shrinkage is a statistical technique that offers a fast alternative to crossvalidation for the regularization of covariance matrices and has appealing consistency properties. We show that the proof of consistency requires bounds on the growth rates of eigenvalues and their dispersion, which are often violated in data. We prove consistency under assumptions which do not restrict the covariance structure and therefore better match real world data. In addition, we propose an extension of analytic shrinkage -orthogonal complement shrinkage-which adapts to the covariance structure. Finally we demonstrate the superior performance of our novel approach on data from the domains of finance, spoken letter and optical character recognition, and neuroscience.
The authors focus on the problem of the estimation of 
      covariance matrices using empirical estimators. To avoid overfitting in 
      high-dimensional cases they shrink the covariance matrix towards a 
      multiple of the identity matrix. The optimal amount of shrinkage \lambda 
      can be estimated empirically by minimizing the expected error of the 
      shrinkage estimate. Then they show that existing results about the 
      consistency of the estimate of \lambda depend on an assumption which does 
      not hold in practice for most real-world datasets. The authors then 
      introduce a new wider set of assumptions and proof that they guarantee 
      consistency. Also, they show that in certain cases there will be no 
      asymptotic shrinkage of the empirical estimate of the covariance matrix. 
      As a solution that they propose is then to perform the shrinkage in the 
      orthogonal complement to the eigen vector with largest eigen value. Since 
      a dataset may have multiple large eigenvalues they iterate the procedure 
      and automatically select the number of retained eigen directions. A series 
      of experiments with synthetic and real-world data validate the proposed 
      approach. Quality: The paper seems to be sound, with the 
      experiments validating the theoretical results. I did not verified the 
      proofs of the main theorems since these are in the supplementary material, 
      which I am not required to examine. The experimental evaluation seems to 
      be thorough. Clarity: The paper is clearly written and 
      well organized. However, the main document does not describe how the 
      orthogonal complement shrinkage is actually performed and the motivation 
      for this approach is also not strong enough in the paper. The authors 
      should clarify how this technique is implemented in practice. 
      Originality: I find interesting and worth on its own the 
      contribution of the authors to show that the existing shrinkage techniques 
      will not produce enough shrinkage in some cases. The proposed orthogonal 
      complement shrinkage is novel up to my knowledge. However, it is a bit 
      incremental since it represents only a modification of already existing 
      approaches. Significance: In my opinion, the most 
      significant contribution is to point out the failure of the shrinkage 
      method in several relevant cases, as well as the theoretical contributions 
      contained in the paper. The orthogonal complement shrinkage seems to be 
      less significant than these other elements.