Exact Post Model Selection Inference for Marginal Screening
We develop a framework for post model selection inference, via marginal screening, in linear regression. At the core of this framework is a result that characterizes the exact distribution of linear functions of the response y, conditional on the model being selected ("condition on selection" framework). This allows us to construct valid confidence intervals and hypothesis tests for regression coefficients that account for the selection procedure. In contrast to recent work in high-dimensional statistics, our results are exact (non-asymptotic) and require no eigenvalue-like assumptions on the design matrix X. Furthermore, the computational cost of marginal regression, constructing confidence intervals and hypothesis testing is negligible compared to the cost of linear regression, thus making our methods particularly suitable for extremely large datasets. Although we focus on marginal screening to illustrate the applicability of the condition on selection framework, this framework is much more broadly applicable. We show how to apply the proposed framework to several other selection procedures including orthogonal matching pursuit and marginal screening+Lasso.
The purpose of this paper is to derive exact confidence intervals for linear regression coefficients post model selection. The authors first introduce the notation of the regression model under study and the marginal screening as one simple algorithm for variable selection. They illustrate empirically that the commonly used z-test interval is not correctly calibrated and thus not valid. The authors then construct the distribution of the target variable conditioned on the selection event, which leads to confidence intervals for, e.g., the marginal screening. Finally, the confidence intervals are studied empirically on two real-world datasets.* Quality: Overall the paper is technically sound. However, it fails to evaluate all strength and weaknesses of the proposed method. Specifically:- In hypothesis testing (confidence intervals), there is always the trade-off between type I and II error. The type I error is, unfortunately, not well-studied. This could be done, e.g., with simulated data where you have control over the ground truths (correlation between features and target label).- You state that the POSI [2] method is computational expensive and thus not applicable for d larger than 30. But that means that you can use POSI in Section 7.1 or study its behaviour on artificial data. - There are missing baselines: What is the performance (power and correctness) compared to other baselines as greedy algorithms, POSI,... mentioned in Section 2?- The depending on the number of dimensions and the coverage rate depending on n would also be worth to be studied. * Clarity: Overall, the paper is comprehensibly structured, but the context is often rarely described, which makes it hard to follow. Furthermore, some key concepts and notation are ambiguous or not introduced. Specifically,- The model is introduced immediately in the first sentence but the notation is actually formally defined in Section 3. At the first part some aspects are incompletely presented. E.g., what is the number of examples? What is \mu (in contrast to \mu(x))?- The notation for the observation and the noise free label is confusing: "(X^TX)^{−1} X^T\mu, which is the best linear predictor of \mu" (line 039) makes not really sense to me. Isn't \mu the best estimator for \mu? Do you mean "(X^TX)^{−1} X^T y" is the best predictor of \mu? The same holds for "\mu = X \beta^0".- \mu(x) is assumed to be arbitrary in the beginning but only the linear case is studied. To clarify things, why don't you restrict yourself to linear models at all?- While the need for variable selection is motivated the need of confidence intervals is not. Can you clarify this in more detail?- 7.1 studies the dependance between empirical and theoretical coverage. However, Section 7.2 present only one outcome of your method on the genomic dataset. What is the significance of that statement? Is that already known and does it confirm the correctness of your approach or is that a new outcome and gives new insight to the community?- "Z-test", "Adjusted",  and "Nominal" in Figure 3 not introduced* Originality: The paper applies the idea of [12] to derive the exact distribution of the target variables by conditioning on the selection event to marginal screening.* Significance: The empirical study fails to evaluate all strength and weaknesses and the theoretical results are quite similar to [12].