title: CLoVe: Encoding Compositional Language in Contrastive Vision-Language
    Models
summary: Recent years have witnessed a significant increase in the performance of
  Vision and Language tasks. Foundational Vision-Language Models (VLMs), such as
  CLIP, have been leveraged in multiple settings and demonstrated remarkable
  performance across several tasks. Such models excel at object-centric
  recognition yet learn text representations that seem invariant to word order,
  failing to compose known concepts in novel ways. However, no evidence exists
  that any VLM, including large-scale single-stream models such as GPT-4V,
  identifies compositions successfully. In this paper, we introduce a framework
  to significantly improve the ability of existing models to encode compositional
  language, with over 10% absolute improvement on compositionality benchmarks,
  while maintaining or improving the performance on standard object-recognition
  and retrieval benchmarks. Our code and pre-trained models are publicly
  available at https://github.com/netflix/clove.
paper_authors._id: 65dbf75b594bab313c713f68
paper_authors.name: Santiago Castro
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-26T04:03:04.956Z
paper_authors._id: 5e7b8bd930dc073f817a2ba2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1628132691344-5e7b8bd930dc073f817a2ba2.jpeg
paper_authors.fullname: Santiago Castro
paper_authors.type: user
paper_authors.user: bryant1410
paper_authors._id: 65dbf75b594bab313c713f69
paper_authors.name: Amir Ziai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T10:44:24.809Z
paper_authors._id: 63a38a7f3618ae17d6649165
paper_authors.avatarUrl: /avatars/1bf374f038f3cf27415f14d07b03d5b7.svg
paper_authors.fullname: Amir
paper_authors.type: user
paper_authors.user: amirziai
paper_authors._id: 65dbf75b594bab313c713f6a
paper_authors.name: Avneesh Saluja
paper_authors._id: 65dbf75b594bab313c713f6b
paper_authors.name: Zhuoning Yuan
paper_authors._id: 65dbf75b594bab313c713f6c
paper_authors.name: Rada Mihalcea
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-26T11:05:13.981Z
paper_authors._id: 64153d8f95fb6f824b22d960
paper_authors.avatarUrl: /avatars/e189e792dbdb0e2a8a04eda3a8c2bcfc.svg
paper_authors.fullname: Rada Mihalcea
paper_authors.type: user
paper_authors.user: mihalcea
paper_title: CLoVe: Encoding Compositional Language in Contrastive Vision-Language
    Models
paper_summary: Recent years have witnessed a significant increase in the performance of
  Vision and Language tasks. Foundational Vision-Language Models (VLMs), such as
  CLIP, have been leveraged in multiple settings and demonstrated remarkable
  performance across several tasks. Such models excel at object-centric
  recognition yet learn text representations that seem invariant to word order,
  failing to compose known concepts in novel ways. However, no evidence exists
  that any VLM, including large-scale single-stream models such as GPT-4V,
  identifies compositions successfully. In this paper, we introduce a framework
  to significantly improve the ability of existing models to encode compositional
  language, with over 10% absolute improvement on compositionality benchmarks,
  while maintaining or improving the performance on standard object-recognition
  and retrieval benchmarks. Our code and pre-trained models are publicly
  available at https://github.com/netflix/clove.
paper_ai_keywords: Vision-Language Models (VLMs)" CLIP object-centric recognition text representations invariant to word order compositional languages GPT-4V compositionality benchmarks standard object-recognition intervention benchmarks Gen4