High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularization
<|reference_start|>High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularization: This paper studies kernel ridge regression in high dimensions under covariate shifts and analyzes the role of importance re-weighting. We first derive the asymptotic expansion of high dimensional kernels under covariate shifts. By a bias-variance decomposition, we theoretically demonstrate that the re-weighting strategy allows for decreasing the variance. For bias, we analyze the regularization of the arbitrary or well-chosen scale, showing that the bias can behave very differently under different regularization scales. In our analysis, the bias and variance can be characterized by the spectral decay of a data-dependent regularized kernel: the original kernel matrix associated with an additional re-weighting matrix, and thus the re-weighting strategy can be regarded as a data-dependent regularization for better understanding. Besides, our analysis provides asymptotic expansion of kernel functions/vectors under covariate shift, which has its own interest.<|reference_end|>
@article{chen2024high-dimensional,
    title={High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent
  Implicit Regularization},
    author={Yihang Chen, Fanghui Liu, Taiji Suzuki, Volkan Cevher},
    journal={arXiv preprint arXiv:2406.03171},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.03171},
    primaryClass={stat.ML cs.LG}
}