Estimation with Norm Regularization
Analysis of non-asymptotic estimation error and structured statistical recovery based on norm regularized regression, such as Lasso, needs to consider four aspects: the norm, the loss function, the design matrix, and the noise model. This paper presents generalizations of such estimation error analysis on all four aspects. We characterize the restricted error set, establish relations between error sets for the constrained and regularized problems, and present an estimation error bound applicable to any norm. Precise characterizations of the bound is presented for a variety of noise models, design matrices, including sub-Gaussian, anisotropic, and dependent samples, and loss functions, including least squares and generalized linear models. Gaussian width, a geometric measure of size of sets, and associated tools play a key role in our generalized analysis.
1) Please give examples of what new regularizers are included in your analysis. What do you cover that is not decomposable, yet yields fast estimation rates? 2) You give several generalizations of Gordon's inequality to correlated Gaussian and subGaussian design. These are only useful if we can compute the Gaussian width of the set. The few sets that we do know the Gaussian width for are sparse vectors or the similar error cone from Bickel et al. There are several examples in [6] that aren't decomposable and you may want to emphasize these norms, since they have known gaussian widths and aren't decomposable.3) Without specific examples of error cones, and their Gaussian widths, and the regularizers, the importance of the results here are not clear. For example, the regularizers covered in this work are the same as those in [6], which is broader than those in Negahban et al. Do you obtain the same rates of convergence for the Lagrangian formulation as the constrained formulation? This is expected, but it would be nice to see a few examples.4) I just came across a paper http://arxiv.org/pdf/1405.1102.pdf , and I think in Theorem 6.3 on page 11 it contains essentially the same results for "Independent anistotropic sub-gaussian designs". The theorem is stated for sets that are cone intersected with sphere, but of course any closed subset of the sphere can be represented like that. Thus it also applies to all subsets of the sphere. It may not be fair to compare against this paper since it was posted only a month before the NIPS submission deadline. On the other hand, the proof in the Tropp paper seems really easy. Can you please comment?The paper by Mendelson et al. 2007 is also relevant, http://link.springer.com/article/10.1007%2Fs00039-007-0618-7Please discuss the relationship to these two papers.