Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuning
<|reference_start|>Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuning: Prompt Tuning has emerged as a prominent research paradigm for adapting vision-language models to various downstream tasks. However, recent research indicates that prompt tuning methods often lead to overfitting due to limited training samples. In this paper, we propose a Cross-modal Aligned Feature Tuning (Craft) method to address this issue. Cross-modal alignment is conducted by first selecting anchors from the alternative domain and deriving relative representations of the embeddings for the selected anchors. Optimizing for a feature alignment loss over anchor-aligned text and image modalities creates a more unified text-image common space. Overfitting in prompt tuning also deteriorates model performance on out-of-distribution samples. To further improve the prompt model's robustness, we propose minimizing Maximum Mean Discrepancy (MMD) over the anchor-aligned feature spaces to mitigate domain shift. The experiment on four different prompt tuning structures consistently shows the improvement of our method, with increases of up to $6.1\%$ in the Base-to-Novel generalization task, $5.8\%$ in the group robustness task, and $2.7\%$ in the out-of-distribution tasks. The code will be available at https://github.com/Jingchensun/Craft<|reference_end|>
@article{sun2024craft:,
    title={Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuning},
    author={Jingchen Sun, Rohan Sharma, Vishnu Suresh Lokhande, Changyou Chen},
    journal={arXiv preprint arXiv:2407.15894},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.15894},
    primaryClass={cs.CV}
}