A Residual Bootstrap for High-Dimensional Regression with Near Low-Rank Designs
We study the residual bootstrap (RB) method in the context of high-dimensional linear regression. Specifically, we analyze the distributional approximation of linear contrasts c ( β ρ − β), where β ρ is a ridge-regression estimator. When regression coefficients are estimated via least squares, classical results show that RB consistently approximates the laws of contrasts, provided that p n, where the design matrix is of size n × p. Up to now, relatively little work has considered how additional structure in the linear model may extend the validity of RB to the setting where p/n 1. In this setting, we propose a version of RB that resamples residuals obtained from ridge regression. Our main structural assumption on the design matrix is that it is nearly low rank -in the sense that its singular values decay according to a power-law profile. Under a few extra technical assumptions, we derive a simple criterion for ensuring that RB consistently approximates the law of a given contrast. We then specialize this result to study confidence intervals for mean response values X i β, where X i is the ith row of the design. More precisely, we show that conditionally on a Gaussian design with near low-rank structure, RB simultaneously approximates all of the laws X i ( β ρ − β), i = 1, . . . , n. This result is also notable as it imposes no sparsity assumptions on β. Furthermore, since our consistency results are formulated in terms of the Mallows (Kantorovich) metric, the existence of a limiting distribution is not required.
The authors propose a theoretical analysis of the residual bootstrap for theridge regression estimator, under general (additive) noise model. The analysis provided requires some structural assumptionthat the empirical covariance of the variables is nearly low rank (in the sensethat its spectrum decreases polynomially fast toward zero). Under some moretechnical assumptions (e.g. the design matrix is Gaussian) the authors prove that the residual bootstrap approximates all the conditional laws of all fitted values.Quality: The paper is very well written and the organization is clear.It gives new insight on the benefit of the residual bootstrap in possibly large dimension without sparsity assumption, even when n \approx p.Clarity: very good.Originality: proof techniques seems rather standard, but the context is new I believe.Significance: this is a good direction to get prediction interval in large dimension.Such a road has still not been carefully explored until recently (apart from few papers,that the authors mentioned). Line by line comments:032: requires imposes ->imposes125: harmonize the way paper are cited (e.g. B&F 1983 is [17]!)131: distribution distribution->distribution133: focus on role-> focus on the role138: Theorem ?? -> provide the correct name (potentially issue with the supplementary version one)157: the bold notation for the conditional sign | if fairly unusual and looks weird.167: \Phi is not defined before, only \Phi_\rho was defined in (4)Summary of the review:175: correct the recurring error: X^\top X+\rho : the first term is a matrix, the second one is a scalar.The identity matrix is missing. This is everywhere in the article / supplementary.289: a variety bounds ->a variety of bounds310/313: can the authors give some examples where such an assumption is realistic in practice?323:that \beta that->that \beta413:a conclusion is missing, no? And some room is available to do so!416: names are duplicatedMost references: please provide the full name when citing the and double check the wrong ones like [6,8,9,10,12] where the "et al." are useless.Note also that if you want to provide first names, do it equally for all authors (sometimes being the same):e.g.: P. J. Bickel and Peter J. Bickel refer to the same authors!441: citation issue again449: idem507/530: the constant could be tracked I guess here. Overall details are a bit loose around here.