CLoVe: Encoding Compositional Language in Contrastive Vision-Language Models
<|reference_start|>CLoVe: Encoding Compositional Language in Contrastive Vision-Language Models: Recent years have witnessed a significant increase in the performance of Vision and Language tasks. Foundational Vision-Language Models (VLMs), such as CLIP, have been leveraged in multiple settings and demonstrated remarkable performance across several tasks. Such models excel at object-centric recognition yet learn text representations that seem invariant to word order, failing to compose known concepts in novel ways. However, no evidence exists that any VLM, including large-scale single-stream models such as GPT-4V, identifies compositions successfully. In this paper, we introduce a framework to significantly improve the ability of existing models to encode compositional language, with over 10% absolute improvement on compositionality benchmarks, while maintaining or improving the performance on standard object-recognition and retrieval benchmarks. Our code and pre-trained models are publicly available at https://github.com/netflix/clove.<|reference_end|>
@article{castro2024clove:,
    title={CLoVe: Encoding Compositional Language in Contrastive Vision-Language
  Models},
    author={Santiago Castro, Amir Ziai, Avneesh Saluja, Zhuoning Yuan, Rada
  Mihalcea},
    journal={arXiv preprint arXiv:2402.15021},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.15021},
    primaryClass={cs.CV cs.CL}
}