Minimizing Embedding Distortion for Robust Out-of-Distribution Performance
<|reference_start|>Minimizing Embedding Distortion for Robust Out-of-Distribution Performance: Foundational models, trained on vast and diverse datasets, have demonstrated remarkable capabilities in generalizing across different domains and distributions for various zero-shot tasks. Our work addresses the challenge of retaining these powerful generalization capabilities when adapting foundational models to specific downstream tasks through fine-tuning. To this end, we introduce a novel approach we call "similarity loss", which can be incorporated into the fine-tuning process of any task. By minimizing the distortion of fine-tuned embeddings from the pre-trained embeddings, our method strikes a balance between task-specific adaptation and preserving broad generalization abilities. We evaluate our approach on two diverse tasks: image classification on satellite imagery and face recognition, focusing on open-class and domain shift scenarios to assess out-of-distribution (OOD) performance. We demonstrate that this approach significantly improves OOD performance while maintaining strong in-distribution (ID) performance.<|reference_end|>
@article{shaked2024minimizing,
    title={Minimizing Embedding Distortion for Robust Out-of-Distribution
  Performance},
    author={Tom Shaked, Yuval Goldman, Oran Shayer},
    journal={arXiv preprint arXiv:2409.07582},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.07582},
    primaryClass={cs.CV}
}