Advancing Prompt Learning through an External Layer
<|reference_start|>Advancing Prompt Learning through an External Layer: Prompt learning represents a promising method for adapting pre-trained vision-language models (VLMs) to various downstream tasks by learning a set of text embeddings. One challenge inherent to these methods is the poor generalization performance due to the invalidity of the learned text embeddings for unseen tasks. A straightforward approach to bridge this gap is to freeze the text embeddings in prompts, which results in a lack of capacity to adapt VLMs for downstream tasks. To address this dilemma, we propose a paradigm called EnPrompt with a novel External Layer (EnLa). Specifically, we propose a textual external layer and learnable visual embeddings for adapting VLMs to downstream tasks. The learnable external layer is built upon valid embeddings of pre-trained CLIP. This design considers the balance of learning capabilities between the two branches. To align the textual and visual features, we propose a novel two-pronged approach: i) we introduce the optimal transport as the discrepancy metric to align the vision and text modalities, and ii) we introduce a novel strengthening feature to enhance the interaction between these two modalities. Four representative experiments (i.e., base-to-novel generalization, few-shot learning, cross-dataset generalization, domain shifts generalization) across 15 datasets demonstrate that our method outperforms the existing prompt learning method.<|reference_end|>
@article{cui2024advancing,
    title={Advancing Prompt Learning through an External Layer},
    author={Fangming Cui, Xun Yang, Chao Wu, Liang Xiao, Xinmei Tian},
    journal={arXiv preprint arXiv:2407.19674},
    year={2024},
    doi={10.1145/3664647.3680953},
    archivePrefix={arXiv},
    eprint={2407.19674},
    primaryClass={cs.CV}
}