Selective inference for group-sparse linear models
We develop tools for selective inference in the setting of group sparsity, including the construction of confidence intervals and p-values for testing selected groups of variables. Our main technical result gives the precise distribution of the magnitude of the projection of the data onto a given subspace, and enables us to develop inference procedures for a broad class of group-sparse selection methods, including the group lasso, iterative hard thresholding, and forward stepwise regression. We give numerical results to illustrate these tools on simulated data and on health record data.
The paper proposes an approach to construct confidence intervals and p-values for the selection of groups of covariates. The approach is applicable to a broad class of methods, such as forward selection, iterative hard-thresholding, and group lasso, which are taken as illustrative examples in the paper. The challenge of the addressed problem is related to the fact that the statistical assessment of the importance of the groups must account for the fact that the same data have been used to select the groups. The novelty of the paper lies in the ability to construct confidence intervals while previous approaches for groups of covariates were limited to p-values. The contribution relies on a key lemma describing the distribution of truncated projections of Gaussian's. Two small-scale experiments are carried out, on synthetic and real-world data.
Clarity/Presentation/Related work/Presentation of contribution(s): The paper is overall clearly and well written. Also, the paper is well structured and makes a good introduction to the problem at hand, exposing what the new challenges and the resulting contributions are (e.g., paragraph lines 103-115). At some few places (see details below), some additional details would be useful.   Technical level: The paper appears as technically sound and Lemma 1/Theorem 1 represent a non-trivial contribution. Some questions related to the proofs (see afterwards) should be clarified.   Experiments: The experimental section may appear as a bit disappointing. For instance, it is unclear what the message of the second experiment (Section 4.2) is? Moreover, some further comparisons/discussions to other approaches may be beneficial (see details below). For example, since the main challenge of this work is to try to work with the same data as those used for the selection of the groups, it would make sense to compare against the more naive strategy based on splitting the data (which is supposed to lead to some loss of accuracy for model selection and power for inference).    Details comments:  * It should be discussed earlier in the paper whether the groups are assumed to form a partition (i.e., no overlaps).  * A discussion about going beyond the quadratic loss (or equivalently, the Gaussian model) would be interesting.  * Condition (1) should be more detailed: Why is this the right quantity to consider?  * In Theorem 1, it would useful to have more details regarding the quantities we condition on and with respect to which random variables the probabilities are considered.  * How is the equation hat{f_Y}(L_alpha) = alpha numerically solved? (line 252)  * A discussion about the computational complexity of the approach in the different scenarios (IHT, group lasso, etc.) would be useful.  * It is unclear to understand what the take-home message of the second experiment (Section 4.2) is?  * Could the proposed methodology be used for techniques like [a, and references therein]? Moreover, how would the proposed approach compare with a simple bootstrap operation (along the lines of [b] but for group-sparse estimators)?  * Supplementary material: In the beginning of the proof of Theorem 1, don't we need as well the continuity of t -> f_y(t) to guarantee the existence of L_alpha? More details should also be provided regarding the proofs of the properties of f_y (e.g., limiting behaviors and monotonicity).  [a] Ndiaye, E.; Fercoq, O.; Gramfort, A. & Salmon, J. GAP Safe screening rules for sparse multi-task and multi-class models Advances in Neural Information Processing Systems, 2015, 811-819 [b] Bach, F. Bolasso: model consistent Lasso estimation through the bootstrap Proceedings of the International Conference on Machine Learning (ICML), 2008