Regularization-Free Estimation in Trace Regression with Symmetric Positive Semidefinite Matrices
Trace regression models have received considerable attention in the context of matrix completion, quantum state tomography, and compressed sensing. Estimation of the underlying matrix from regularization-based approaches promoting low-rankedness, notably nuclear norm regularization, have enjoyed great popularity. In this paper, we argue that such regularization may no longer be necessary if the underlying matrix is symmetric positive semidefinite (spd) and the design satisfies certain conditions. In this situation, simple least squares estimation subject to an spd constraint may perform as well as regularization-based approaches with a proper choice of regularization parameter, which entails knowledge of the noise level and/or tuning. By contrast, constrained least squares estimation comes without any tuning parameter and may hence be preferred due to its simplicity.
The paper studies the estimation of low-rank semidefinite matrices from noisy inner products with measurement matrices X_1, ..., X_n. Nuclear norm regularized least squares is the dominant approach to estimation in this setting. However, if \Sigma is forced to be positive semidefinite, the nuclear norm reduces to the trace (a single linear function). The paper proves that under some circumstances, one can accurately estimate \Sigma by simply solving a constrained least squares problem (dropping this linear regularization). This is not possible for general Gaussian measurements, but for measurements that do not have semidefinite matrices in their nullspace (e.g., if the X_i are themselves full rank semidefinite matrices), it is, under conditions. Simulations verify the claims. The paper also presents a real data experiment, which appears to be somewhat artificial, but again agrees with the theory presented here. The goal of the paper is to argue that regularization may not be necessary for certain sampling models. How do the theoretical results compare to the best known theoretical results for the regularized problem? The restricted eigenvalue condition is necessary for both formulations; it seems that the separability and correlation conditions are only necessary for the constrained least squares. Comments in the response would be helpful. Suppose we specialize the results to some random measurement ensemble (e.g., X_i = z_i z_i^*, with z_i a random vector, or X_i a Wishart matrix). Is it possible to evaluate \mu, \tau and \phi, and compare the bound in Theorem 2 to the corresponding result for the nuclear norm regularized variant (11)? Comments in the response would be helpful. In a similar vein, while it is easy to recognize that the results in this paper are more general than those of [4]/[10], it would be great if we could recover the rates of the special case as a consequence of the general theory introduced here. A comparison would help to understand the strength of the results. Comments in the response would be appreciated. The main practical implication of the work seems to be that in some situations, it is not necessary to choose a regularization parameter. From a computational perspective, there don't seem to be obvious reasons to prefer (7) to (11). The paper contains solid results, which seem somewhat under-interpreted. I'm eager to hear the authors responses on the above points, as well as some questions below about the practical motivations for the sampling model.  EDIT: after considering the authors responses on the comparison to nuclear norm minimization and the motivations for the sampling model, I have raised my score to a 6.  Smaller issues:Can \mu(T) be controlled in terms of \phi(T) and \tau(T)? A similar correlation quantity shows up in the intermediate steps of classical compressed sensing proofs, where it is bounded in terms of restricted isometry constants. The paper contains a real data experiment showing the estimation of the best rank-r approximation of data covariance, based on inner products with random Wishart matrices. The observations seem to conform (more or less) to the theoretical claims of the paper. However, I'm a little bit puzzled as to the motivation for this experiment. There does not seem to be much practical motivation for this kind of sampling with this kind of data. The message of the paper would be much stronger if it could demonstrate the performance of the proposed method in the context of an application where we really are stuck with a few trace samples. The results require special measurements which do not have semidefinite matrices in their nullspace (in a quantitative sense). "Generic" gaussian measurements do not satisfy this property, but Wishart measurements do. Besides phase retrieval (where this is a somewhat natural model) are there other practical scenarios where trace sampling with semidefinite X_i applies?