Compressive spectral embedding: sidestepping the SVD
Spectral embedding based on the Singular Value Decomposition (SVD) is a widely used "preprocessing" step in many learning tasks, typically leading to dimensionality reduction by projecting onto a number of dominant singular vectors and rescaling the coordinate axes (by a predefined function of the singular value). However, the number of such vectors required to capture problem structure grows with problem size, and even partial SVD computation becomes a bottleneck. In this paper, we propose a low-complexity compressive spectral embedding algorithm, which employs random projections and finite order polynomial expansions to compute approximations to SVD-based embedding. For an m×n matrix with T non-zeros, its time complexity is O ((T + m + n) log(m + n)), and the embedding dimension is O(log(m + n)), both of which are independent of the number of singular vectors whose effect we wish to capture. To the best of our knowledge, this is the first work to circumvent this dependence on the number of singular vectors for general SVD-based embeddings. The key to sidestepping the SVD is the observation that, for downstream inference tasks such as clustering and classification, we are only interested in using the resulting embedding to evaluate pairwise similarity metrics derived from the ℓ 2 -norm, rather than capturing the effect of the underlying matrix on arbitrary vectors as a partial SVD tries to do. Our numerical results on network datasets demonstrate the efficacy of the proposed method, and motivate further exploration of its application to large-scale inference tasks.
The authors propose a novel algorithm for computing "compressive embeddings" of a matrix, which is most often done via approximate partial SVD. They provide detailed motivation and background for the problem, reviewing applications in graph processing, NLP, etc. The proposed algorithm "side-steps" the SVD by leveraging the JL lemma to compute approximate pair-wise distances and polynomial approximations to incorporate approximate functions of the singular values (e.g., thresholds). Some implementation details are highlighted.Overall, I think this a very good paper that presents simple, fast, effective, and (to my knowledge) novel algorithm for computing compressive embeddings, which is likely to be superior to Randomized SVD and pure random projections for many applications. The presentation is also relatively clear, if dense, despite a few minor errors and formatting issues (outlined below). One issue is that the authors do not flesh out a detailed story of how the performance changes relative to the embedding dimension and types of polynomials that might be approximated. This is difficult due to space-constraints, but especially for the former it would have been helpful to have a plot relating accuracy to dimension (not just a comparison btw two dimension sizes) that essentially provided an empirical view of theorem 1. Moreover, since the top 500 eigenvalues were computed the \delta could be reasonably approximated and so an empirical vs. theoretical expectation plot is possible.  Related to the above, more concrete recommendations from the authors w.r.t. how to choose embedding dimensions and eigenvalue thresholds would be helpful. And it is somewhat misleading that the authors set the polynomial approximations given knowledge of the first 500 eigenvalues. Discussion of what to do when such prior computations are infeasible would be helpful. Minor issues:-\cite{foo}\cite{bar} -> \cite{foo,bar} everywhere - line 73: point (a) is unclear - line 113: iteration\cite{foo} -> iteration \cite{foo} - line 237: "of good" -> "of a good" - line 340: "(=" -> "=" - capitalize Euclidean, Hermitian etc.