Model Selection for High-Dimensional Regression under the Generalized Irrepresentability Condition
In the high-dimensional regression model a response variable is linearly related to p covariates, but the sample size n is smaller than p. We assume that only a small subset of covariates is 'active' (i.e., the corresponding coefficients are non-zero), and consider the model-selection problem of identifying the active covariates. A popular approach is to estimate the regression coefficients through the Lasso ( 1 -regularized least squares). This is known to correctly identify the active set only if the irrelevant covariates are roughly orthogonal to the relevant ones, as quantified through the so called 'irrepresentability' condition. In this paper we study the 'Gauss-Lasso' selector, a simple two-stage method that first solves the Lasso, and then performs ordinary least squares restricted to the Lasso active set. We formulate 'generalized irrepresentability condition' (GIC), an assumption that is substantially weaker than irrepresentability. We prove that, under GIC, the Gauss-Lasso correctly recovers the active set.
This paper provides a thorough and comprehensive study 
      of the post-Lasso (the estimator obtained by fitting a least 
      square model on the variables selected by the Lasso) in the context of 
      high dimensional sparse regression. A new theoretical guarantee is 
      provided for the post- Lasso, as well as simulated experiments on a 
      toy example where the benefit of the procedure is clear. 
      Though, there are still a few points that could be improved. 
      First, some important references are missing about other works 
      considering the post-Lasso on a theoretical level (or a variation 
      of it): "Pivotal Estimation of Nonparametric Functions via 
      Square-root Lasso" Alexandre Belloni, Victor Chernozhukov, Lie Wang 
      (cf. Theorem 7 for instance) "L1-Penalized Quantile 
      Regression in High-Dimensional Sparse Models", Alexandre Belloni and 
      Victor Chernozhukov, 2011 (cf. post-l1-QR Theorem 5) 
      Moreover, a recent work focusing on the practical 
      properties of the post-Lasso for a particular set of simulated data, 
      challenges, in certain situations, the theoretical benefit illustrated 
      by the authors. Can they comment on that? "Trust, but verify: 
      benefits and pitfalls of least-squares refitting in high dimensions",2013 
      Johannes Lederer Points to correct: -l134: 
      without further assumptions the minimizers of G might not be unique. 
      the results is true under some more assumptions on the Gramm matrix, 
      as is well known (and in a way proved later by the authors) since 
      "On Sparse Representations in Arbitrary Redundant Bases", J-J. 
      Fuchs,2004 and more recently "The Lasso Problem and Uniqueness", 
      Ryan J. Tibshirani, 2013 The problem occurs many time in the 
      proof: the unicity is sometimes used before it is proved. Adapting the 
      results from the aforementioned papers, I encourage the authors to 
      show that (under suitable assumptions) unicity holds (cf. for instance 
      l1034, where the strict inequality is given without any justification, 
      see also l1071 and l1113) and therefore that there proof is right. 
      I encourage the authors to fix this for the sake of clarity: it 
      could also be better to add an assumption mentioning when one needs 
      the (correctly) extracted Gramm matrix to be invertible. 
      -l303: t_0 is defined but nowhere used in this section, and 
      then re-used in the next one... please remove. -l307: A comment 
      could be added on the fact that the lambda parameter depends on an unknown 
      quantity, eta. -l317: What is the benefit of the assumptions w.r.t 
      [23]? It does not seem straightforward which one is weaker: on the one 
      hand we need a matrix invertible of larger size (T_0 contains S) but on 
      the other hand only the sup-norm of a vector should be control is the 
      proposed work. -Section 3.1 is for me useless, the results are 
      exactly the same as in Section 2.1. Please remove and spare some room 
      for the comments, and the missing references. For instance, further 
      comparisons on the differences between the deterministic case and the 
      random case could be investigated. -l408: Did the authors try to 
      improve the term depending on exp(-t_0/2)? It seems to be the weak 
      part of the probability control provided in Theorem 3.4 -l441/443: 
      Candes and Cand\'es are mispelled. It should be Cand\`es -l553: it 
      seems there is a sign issue. General questions: - can 
      the authors comment on the fact that the sparsity level must be known a 
      prior (no adaptivity) in their procedure? -When is T_0=S (l267)? That 
      would be interesting to understand when the two are identical.