Matrix Norm Estimation from a Few Entries
Singular values of a data in a matrix form provide insights on the structure of the data, the effective dimensionality, and the choice of hyper-parameters on higher-level data analysis tools. However, in many practical applications such as collaborative filtering and network analysis, we only get a partial observation. Under such scenarios, we consider the fundamental problem of recovering various spectral properties of the underlying matrix from a sampling of its entries. We propose a framework of first estimating the Schatten k-norms of a matrix for several values of k, and using these as surrogates for estimating spectral properties of interest, such as the spectrum itself or the rank. This paper focuses on the technical challenges in accurately estimating the Schatten norms from a sampling of a matrix. We introduce a novel unbiased estimator based on counting small structures in a graph and provide guarantees that match its empirical performances. Our theoretical analysis shows that Schatten norms can be recovered accurately from strictly smaller number of samples compared to what is needed to recover the underlying low-rank matrix. Numerical experiments suggest that we significantly improve upon a competing approach of using matrix completion methods.
Summary of paper:
The paper proposes an estimator for the Schatten norm of a matrix when only few entries of the matrix is observed. The estimator is based on the relation between the Schatten k-norm of a matrix and the total weight of all closed k-length walks on the corresponding weighted graph. While the paper restricts the discussion to symmetric positive semi-definite matrices, and guarantees given for uniform sampling, the generic principle is applicable for any matrix and any sampling strategy. The results hold under standard assumptions, and corollaries related to spectrum and rank estimation are also provided in the appendix.
Clarity:
The paper is very well written and organised, including some of the material in the supplementary. The motivation for the work, and the discussion that builds up to the estimator and well presented. The numerical simulations are also perfectly positioned.
Quality:
The technical quality of the paper is high. The main structure of the proofs are correct, but I didn’t verify the computations in equations (26)-(36).
The most pleasing component of the theoretical analysis is the level at which it is presented. The authors restrict the discussion to symmetric psd matrices for ease of presentation, but the sampling distribution is kept general until the final guarantees in Section 3.1. The authors also correctly note estimating the Schatten norm itself is never the end goal, and hence, they provide corollaries related to spectrum and rank estimation in the appendix. The numerical simulations do not involve a real application, but clearly show the advantage of the proposed estimator over other techniques when sampling rate is very small. That being said, I feel that further numerical studies based on this approach is important.
Significance:
The paper considers a classical problem that lies at the heart of several matrix related problems. Hence, the paper is very relevant for the machine learning community. The generality at which the problem is solved, or its possible extensions, has several potential applications.
To this end, I feel that it will be great if the authors can follow up this work with a longer paper that also includes the more complicated case of rectangular matrices, analyses other sampling strategies (like column sampling / sub matrix sampling), and more thorough experimental studies.
Originality:
Since the underlying problem is classical, one should consider that many ideas have been tried out in the literature. I feel that the individual components of the discussion and analysis are fairly standard techniques (except perhaps the computational simplification for k<=7). But I feel that the neat way in which the paper combines everything is very original. For instance, the use of polynomial concentration in the present context is intuitive, but the careful computation of the variation bound is definitely novel.
Minor comments:
1. Line 28-29: Can we hope to estimate the spectrum of a matrix from a submatrix? I am not very optimistic about this.
2. Appendix B: It would be nice if a description of this method is added. In particular, what is the time complexity of the method?
3. Line 238,262,281,292: In standard complexity terminology, I think d^2p = O(something) should actually be d^2p = \Omega(something). Big-O usually means at most upto constant, whereas here I think you would like to say that the number of observed entries is at least something (Big-Omega).
4. Line 263: How do you get the condition on r? The third term in (8) is less than 1 if r is less than d^(1-1/k).