Compositional Kronecker Context Optimization for Vision-Language Models
<|reference_start|>Compositional Kronecker Context Optimization for Vision-Language Models: Context Optimization (CoOp) has emerged as a simple yet effective technique for adapting CLIP-like vision-language models to downstream image recognition tasks. Nevertheless, learning compact context with satisfactory base-to-new, domain and cross-task generalization ability while adapting to new tasks is still a challenge. To tackle such a challenge, we propose a lightweight yet generalizable approach termed Compositional Kronecker Context Optimization (CK-CoOp). Technically, the prompt's context words in CK-CoOp are learnable vectors, which are crafted by linearly combining base vectors sourced from a dictionary. These base vectors consist of a non-learnable component obtained by quantizing the weights in the token embedding layer, and a learnable component constructed by applying Kronecker product on several learnable tiny matrices. Intuitively, the compositional structure mitigates the risk of overfitting on training data by remembering more pre-trained knowledge. Meantime, the Kronecker product breaks the non-learnable restrictions of the dictionary, thereby enhancing representation ability with minimal additional parameters. Extensive experiments confirm that CK-CoOp achieves state-of-the-art performance under base-to-new, domain and cross-task generalization evaluation, but also has the metrics of fewer learnable parameters and efficient training and inference speed.<|reference_end|>
@article{ding2024compositional,
    title={Compositional Kronecker Context Optimization for Vision-Language Models},
    author={Kun Ding and Xiaohui Li and Qiang Yu and Ying Wang and Haojian Zhang
  and Shiming Xiang},
    journal={arXiv preprint arXiv:2403.11631},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.11631},
    primaryClass={cs.CV}
}