Confidence Intervals and Hypothesis Testing for High-Dimensional Statistical Models
Fitting high-dimensional statistical models often requires the use of non-linear parameter estimation procedures. As a consequence, it is generally impossible to obtain an exact characterization of the probability distribution of the parameter estimates. This in turn implies that it is extremely challenging to quantify the uncertainty associated with a certain parameter estimate. Concretely, no commonly accepted procedure exists for computing classical measures of uncertainty and statistical significance as confidence intervals or p-values. We consider here a broad class of regression problems, and propose an efficient algorithm for constructing confidence intervals and p-values. The resulting confidence intervals have nearly optimal size. When testing for the null hypothesis that a certain parameter is vanishing, our method has nearly optimal power. Our approach is based on constructing a 'de-biased' version of regularized Mestimators. The new construction improves over recent work in the field in that it does not assume a special structure on the design matrix. Furthermore, proofs are remarkably simple. We test our method on a diabetes prediction problem.
CONTENT The paper proposes a new method to 
      constructing confidence intervals (CI's) and to perform testing of 
      individual parameters in high-dimensional problems of supervised 
      learning under the sparsity assumption. Although the main focus is on 
      the model of linear regression, an extension to more general cases 
      is presented. A numerical illustration of the proposed methodology is 
      carried out on the diabetes data. The main observation underlying 
      this work is that in the high-dimensional Gaussian linear model, the 
      LSE is inconsistent but unbiased and has a Gaussian distribution with 
      a well-known covariance matrix. Thus it can be used for constructing 
      CI's. On the other hand, the Lasso is consistent under suitable 
      assumptions but has a relatively large bias and, therefore, cannot 
      be used for constructing CI's. The solution, that has been already 
      evoked in previous work by Javanmard and Montanari [9] and van de 
      Geer et al. [23], is to construct CI's based on an estimator which is 
      between the LSE and the Lasso, which has a bias that is small as 
      compared to the variance, under suitable assumptions. The most 
      important question here is which assumptions may be considered as 
      ``acceptable'' in this context. Basically, one can simply use the LSE 
      \hat\theta = pinv(X^T*X)*X^T*Y, where pinv(M) is the pseudo-inverse of 
      a matrix M, and construct a CI based on it. The length of the 
      corresponding CI for the j-th parameter will be 
      \sigma*sqrt(pinv(X^T*X)_{jj}). Under some (very strong) assumptions, 
      this value is of the order 1/sqrt(n) and everything is fine, even in 
      high-dimensions. For instance, this can be done if X is an orthogonal 
      matrix (for p=n). The proposal made by the authors combines 
      the ideas of the aforementioned works with those of [Cai et al., 
      Estimating Sparse Precision Matrix: Optimal Rates of Convergence and 
      Adaptive Estimation, 2012] (cf lemma 1). Recall that Javanmard and 
      Montanari [9] and van de Geer et al. [23] proposed to take the 
      ``intermediate'' estimator in the form hat\theta^u = Lasso + M * X^T * 
      Residual(Lasso) where M is an estimator of the inverse covariance 
      matrix Sigma of the design (one should take this matrix if it is 
      available). An important remark made by the authors of the paper under 
      review is that the only property required from M in order to make the 
      theory work is the convergence to zero of |M*X^T*X - I|_infty. 
      Furthermore, the length of the CI's being proportional to 
      [M*X^T*M^T]_{ii}, the authors propose the smart idea of choosing M by 
      solving a simple optimization problem that minimizes the largest 
      length [M*X^T*M^T]_{ii} under the constraint that |M*X^T*X - I|_infty 
      is bounded by a given constant. EVALUATION pros - the 
      paper reads well - the mathematical results are correct - the idea 
      for choosing M is original and, in my opinion, this choice is better 
      than those proposed in previous works cons - it should be better 
      explained why the proposed approach is better than the simple one 
      based on the LSE. Just saying that X is rank deficient is not enough, 
      since the pseudo-inverse of a rank deficient matrix may still have 
      nonzero entries on the diagonal. - there are some claims in the 
      introduction that over-evaluate the contributions as compared to the 
      previous work. For instance, line 53, the authors claim that they use 
      essentially no condition beyond the standard ones for consistency of 
      the Lasso. This is not true, since among the assumptions they make, I read 
      s_0=o(sqrt(n)/log(p)) [line 208] and norm(Sigma^{-1})_\inf = 
      o(sqrt(n/log(p))), which are much stronger than those required for the 
      Lasso to be consistent. Similarly, lines 128-129, the authors' claim 
      that van de Geer et al. [23] assume that n=Omega(s_0^2\log p) can be 
      often interpreted as the results of the paper under review do not need 
      such a restrictive assumption (which is not true). - The 
      assumption norm(Sigma^{-1})_\inf = o(sqrt(n/log(p))) seems quite strong. 
      Some discussion of it is necessary. - The choice of the parameter 
      gamma, which may be very hard since it will depend on several unknown 
      parameters (such as the largest diagonal entries of Sigma and its 
      inverse) is not discussed. - [minor, can be easily fixed during the 
      revision] some important references are missing: * there are two 
      papers on the CI's in the sparsity setting: - Gautier and Tsybakov 
      [arxiv.org/pdf/1105.2454.pdf] - Nickl and van de Geer 
      [arxiv.org/pdf/1209.1508] * for the scaled Lasso quoted on lines 220 
      and 225, the earlier work by Belloni et al. (Biometrika 2011) should 
      be cited. - [minor, can be easily fixed during the revision] Some 
      mathematical claims lack rigor. Here is a non exhaustive list * line 
      77: it should be stressed that W and X are independent * line 182: the 
      authors define the meaning of ``w.h.p.'', by the convergence of the 
      probability to zero as n\to\infty. However, in the high-dimensional 
      setting, n is not the only quantity that tends to infinity. So the 
      authors should make precise what happens with p and s_0. Are they fix 
      or do they go infinity with n? I guess that the results of the paper 
      require p to go to infinity. This should be stated explicitly early in 
      the text. * line 199: I did not manage to find any clear bound on 
      (M_*SigmaM_*^T)_{ii} in Section 3.2. Please be more precise in the 
      reference. * line 201: I do not agree that Thm 2.1 does not make any 
      assumption on theta_0. In fact, thm 2.1 says that X satisfies the 
      compatibility assumption with S the support of theta_0. If this 
      support has more than n elements, this may not be true. So thm 2.1 
      contains a contraint on the support of theta_0. * Eq (6) is a 
      catastrophe from a mathematical point of view in terms of writing. 
      First, the left hand side is a random variable, while the right hand 
      side is deterministic. Second, since everything depends on x, which 
      varies in R, it is important to explain if o(1) is uniform in x or 
      not.