On the Power of Truncated SVD for General High-rank Matrix Estimation Problems
We show that given an estimate � A that is close to a general high-rank positive semidefinite (PSD) matrix A in spectral norm (i.e., � � A−A� 2 ≤ δ), the simple truncated Singular Value Decomposition of � A produces a multiplicative approximation of A in Frobenius norm. This observation leads to many interesting results on general high-rank matrix estimation problems:
1. High-rank matrix completion: we show that it is possible to recover a general high-rank matrix A up to (1 + ε) relative error in Frobenius norm from partial observations, with sample complexity independent of the spectral gap of A. 2. High-rank matrix denoising: we design an algorithm that recovers a matrix A with error in Frobenius norm from its noise-perturbed observations, without assuming A is exactly low-rank. 3. Low-dimensional approximation of high-dimensional covariance: given N i.i.d. samples of dimension n from N n (0, A), we show that it is possible to approximate the covariance matrix A with relative error in Frobenius norm with N ≈ n, improving over classical covariance estimation results which requires N ≈ n 2 .
This paper shows that given an estimate that is close to the target high-rank PSD matrix A in spectral norm a simple truncated SVD gives an estimate of A that is also close to A in Frobenius norm as well. This leads to a number of applications: high-rank matrix completion, high-rank matrix denoising and low-dimensional estimation of high-dimensional covariance. The paper also gives a bound that depends on the sigma_k - sigma_{k + 1} gap (Theorem 2.2).
Overall, I think this is a pretty strong paper that gives a nice set of results. Specific discussion of some of the strengths and weaknesses is below.
Strengths:
-- Clean set of results, good presentation and clear comparison with the previous work.
-- The algorithm is very simple and is very practical.
Weaknesses:
-- The constant in multiplicative error depends on k and delta
-- Results for high-rank matrix completion are conditional on the matrix satisfying a mu_0-spikiness condition which I didn’t find very natural
-- No experimental results are given
Technical comments:
-- In theorem 2.1 you need a bound on delta in terms of \epsilon and \sigma_{k + 1}. How do you ensure such a bound without knowing sigma_{k + 1}?