Feature Protection For Out-of-distribution Generalization
<|reference_start|>Feature Protection For Out-of-distribution Generalization: With the availability of large pre-trained models, a modern workflow for building real-world machine learning solutions is to fine-tune such models on a downstream task with a relatively small domain-specific dataset. In such applications, one major challenge is that the small fine-tuning dataset does not have sufficient coverage of the distribution encountered when the model is deployed. It is thus important to design fine-tuning methods that are robust to out-of-distribution (OOD) data that are under-represented by the training data. This paper compares common fine-tuning methods to investigate their OOD performance and demonstrates that standard methods will result in a significant change to the pre-trained model so that the fine-tuned features overfit the fine-tuning dataset. However, this causes deteriorated OOD performance. To overcome this issue, we show that protecting pre-trained features leads to a fine-tuned model more robust to OOD generalization. We validate the feature protection methods with extensive experiments of fine-tuning CLIP on ImageNet and DomainNet.<|reference_end|>
@article{tan2024feature,
    title={Feature Protection For Out-of-distribution Generalization},
    author={Lu Tan, Huei Zhou, Yinxiang Huang, Zeming Zheng, Yujiu Yang},
    journal={arXiv preprint arXiv:2405.16027},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.16027},
    primaryClass={cs.LG}
}