ROTI-GCV: Generalized Cross-Validation for right-ROTationally Invariant Data
<|reference_start|>ROTI-GCV: Generalized Cross-Validation for right-ROTationally Invariant Data: Two key tasks in high-dimensional regularized regression are tuning the regularization strength for good predictions and estimating the out-of-sample risk. It is known that the standard approach -- $k$-fold cross-validation -- is inconsistent in modern high-dimensional settings. While leave-one-out and generalized cross-validation remain consistent in some high-dimensional cases, they become inconsistent when samples are dependent or contain heavy-tailed covariates. To model structured sample dependence and heavy tails, we use right-rotationally invariant covariate distributions - a crucial concept from compressed sensing. In the common modern proportional asymptotics regime where the number of features and samples grow comparably, we introduce a new framework, ROTI-GCV, for reliably performing cross-validation. Along the way, we propose new estimators for the signal-to-noise ratio and noise variance under these challenging conditions. We conduct extensive experiments that demonstrate the power of our approach and its superiority over existing methods.<|reference_end|>
@article{luo2024roti-gcv:,
    title={ROTI-GCV: Generalized Cross-Validation for right-ROTationally Invariant
  Data},
    author={Kevin Luo, Yufan Li, Pragya Sur},
    journal={arXiv preprint arXiv:2406.11666},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.11666},
    primaryClass={math.ST cs.LG stat.ML stat.TH}
}