Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrum
<|reference_start|>Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrum: We derive new bounds for the condition number of kernel matrices, which we then use to enhance existing non-asymptotic test error bounds for kernel ridgeless regression (KRR) in the over-parameterized regime for a fixed input dimension. For kernels with polynomial spectral decay, we recover the bound from previous work; for exponential decay, our bound is non-trivial and novel. Our contribution is two-fold: (i) we rigorously prove the phenomena of tempered overfitting and catastrophic overfitting under the sub-Gaussian design assumption, closing an existing gap in the literature; (ii) we identify that the independence of the features plays an important role in guaranteeing tempered overfitting, raising concerns about approximating KRR generalization using the Gaussian design assumption in previous literature.<|reference_end|>
@article{cheng2024characterizing,
    title={Characterizing Overfitting in Kernel Ridgeless Regression Through the
  Eigenspectrum},
    author={Tin Sum Cheng and Aurelien Lucchi and Anastasis Kratsios and David
  Belius},
    journal={arXiv preprint arXiv:2402.01297},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.01297},
    primaryClass={cs.LG stat.ML}
}