Sparse PCA via Covariance Thresholding
In sparse principal component analysis we are given noisy observations of a lowrank matrix of dimension n × p and seek to reconstruct it under additional sparsity assumptions. In particular, we assume here that the principal components v 1 , . . . , v r have at most k 1 , · · · , k q non-zero entries respectively, and study the high-dimensional regime in which p is of the same order as n.
In an influential paper, Johnstone and Lu [JL04] introduced a simple algorithm that estimates the support of the principal vectors v 1 , . . . , v r by the largest entries in the diagonal of the empirical covariance. This method can be shown to succeed with high probability if k q ≤ C 1 n/ log p, and to fail with high probability if k q ≥ C 2 n/ log p for two constants 0 < C 1 , C 2 < ∞. Despite a considerable amount of work over the last ten years, no practical algorithm exists with provably better support recovery guarantees. Here we analyze a covariance thresholding algorithm that was recently proposed by Krauthgamer, Nadler and Vilenchik [KNV13]. We confirm empirical evidence presented by these authors and rigorously prove that the algorithm succeeds with high probability for k of order √ n. Recent conditional lower bounds [BR13] suggest that it might be impossible to do significantly better. The key technical component of our analysis develops new bounds on the norm of kernel random matrices, in regimes that were not considered before.
The authors affirm a conjecture posed by Krauthgamer, Nadler and Vilenchik. regarding the performance of an algorithm for sparse PCA. Namely, they consider doing sparse PCA by (essentially) performing PCA on a thresholded sample covariance matrix. They show that the algorithm succeeds in recovering the joint support of the sparse components up to the threshold k = O(\sqrt(n)), where "k" is the sparsity index and "n" the sample size. This is believed to be the threshold achievable by polynomial-time algorithms. The result closes the gap k \in [\sqrt{n/log p}, \sqrt{n}] by showing an efficient algorithm that works in this regime.I find the results interesting and the paper well-written, modulo some typos and inconsistencies which I assume are introduced from translating a longer version to NIPS format.  I have a few concerns and questions:(1) The authors focus on recovering the union of the supports of the components in the multi-spiked case. First, this is not clear from the start until presented in the theorems and discussed in Remark 2.1. It is worth emphasizing this early on. I also don't quite agree with the content of Remark 2.1. The authors mention that given the joint support and "n" fresh samples, it is possible to consistently estimate the individual supports, because one can get consistent estimates of the individual components (v_q) once one restrict to the joint support. I agree that one can get consistency for v_q's by usual PCA restricted to joint support, but this is \ell_2 consistency. It is not clear whether thresholding these eigenvectors will give the correct supports. In particular, the problem seems to depend on the relative sizes of the supports and the number of components "r". (Consider many components having very small and very large supports).(2) The second half of assumption (A2) is a bit questionable as the authors point out later in Remark 2.3. It is better to move that remark closer to where the assumption is presented. I have no serious objection here, as the results are interesting even in the single component case. However, I want to point out that by symmetry (switching q and q'), the second half of assumption is effectively assuming |v_{q,i}/v_{q',i}| = \gamma. That is, the reverse inequality also holds by assumption.(3) Is the passage of the estimated eigenvector once more through the estimated covariance matrix really necessary? This is step (7) of Algorithm 1. What seems to be implied is that without it consistency is not achieved. In particular, merely thresholding the eigenvectors does not provide the correct supports. Is this true or is the analysis inconclusive in this regard? Any comments clarifying this issue is helpful. Also, you might point out that you are looking at a slightly modified version of what was proposed by Krauthgamer, Nadler and Vilenchik.Some minor issues:- Algorithm (1) appears before some of the quantities involved are introduced. It is better to move it further down.- p.3, in step 3 of rough algorithm, B should be defined in terms of \hat{v}_1 and not the population version v_1.- Algorithm 1, definition of G': the upper limit of the sum should be 2n.- p.4, Table 1 should be replaced with Algorithm 1.- p.4, ... provides bounds "on" the estimation error.- p.5, ... "a" kernel inner product random matrix.- p.5, in the first displayed equation \tilde{z}_i is not defined.