On the Generalization Ability of Unsupervised Pretraining
<|reference_start|>On the Generalization Ability of Unsupervised Pretraining: Recent advances in unsupervised learning have shown that unsupervised pre-training, followed by fine-tuning, can improve model generalization. However, a rigorous understanding of how the representation function learned on an unlabeled dataset affects the generalization of the fine-tuned model is lacking. Existing theoretical research does not adequately account for the heterogeneity of the distribution and tasks in pre-training and fine-tuning stage. To bridge this gap, this paper introduces a novel theoretical framework that illuminates the critical factor influencing the transferability of knowledge acquired during unsupervised pre-training to the subsequent fine-tuning phase, ultimately affecting the generalization capabilities of the fine-tuned model on downstream tasks. We apply our theoretical framework to analyze generalization bound of two distinct scenarios: Context Encoder pre-training with deep neural networks and Masked Autoencoder pre-training with deep transformers, followed by fine-tuning on a binary classification task. Finally, inspired by our findings, we propose a novel regularization method during pre-training to further enhances the generalization of fine-tuned model. Overall, our results contribute to a better understanding of unsupervised pre-training and fine-tuning paradigm, and can shed light on the design of more effective pre-training algorithms.<|reference_end|>
@article{deng2024on,
    title={On the Generalization Ability of Unsupervised Pretraining},
    author={Yuyang Deng, Junyuan Hong, Jiayu Zhou, Mehrdad Mahdavi},
    journal={arXiv preprint arXiv:2403.06871},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.06871},
    primaryClass={cs.LG stat.ML}
}