title: COLA: How to adapt vision-language models to Compose Objects Localized
    with Attributes?
summary: Compositional reasoning is a hallmark of human visual intelligence; yet
  despite the size of large vision-language models, they struggle to represent
  simple compositions by combining objects with their attributes. To measure this
  lack of compositional capability, we design Cola, a text-to-image retrieval
  benchmark to Compose Objects Localized with Attributes. Using Cola as a
  testbed, we explore modeling designs to adapt pre-trained vision-language
  models to reason compositionally about multiple attributes attached to multiple
  objects. We explore 6 finetuning strategies on 2 seminal vision-language
  models, using 3 finetuning datasets and 2 test benchmarks (Cola and CREPE).
  Surprisingly, our optimal finetuning strategy improves a 151M parameter CLIP,
  which disjointly encodes image and language during pretraining, to perform as
  well as a 241M parameter FLAVA, which uses a multi-modal transformer encoder
  during pretraining to attend over both vision and language modalities. This
  optimal finetuning strategy is a lightweight multi-modal adapter that jointly
  attends over both image and language features generated by the pretrained
  model. We show this works better than common strategies such as
  prompt/fine-tuning, or tuning a comparable number of unimodal layers.
paper_authors._id: 64585a86aa426bae5e07231a
paper_authors.name: Arijit Ray
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-08T16:08:50.617Z
paper_authors._id: 6303f2f01dd5d3c6248559ae
paper_authors.avatarUrl: /avatars/17fc6c2f1db9fce9ac9eeeac1cfbad31.svg
paper_authors.fullname: Arijit
paper_authors.type: user
paper_authors.user: array
paper_authors._id: 64585a86aa426bae5e07231b
paper_authors.name: Filip Radenovic
paper_authors._id: 64585a86aa426bae5e07231c
paper_authors.name: Abhimanyu Dubey
paper_authors._id: 64585a86aa426bae5e07231d
paper_authors.name: Bryan A. Plummer
paper_authors._id: 64585a86aa426bae5e07231e
paper_authors.name: Ranjay Krishna
paper_authors._id: 64585a86aa426bae5e07231f
paper_authors.name: Kate Saenko
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-05T10:07:05.146Z
paper_authors._id: 64414f17ad24e9b2cfbd5c29
paper_authors.avatarUrl: /avatars/9e527ef654424d96e5c29d9f76979341.svg
paper_authors.fullname: K S
paper_authors.type: user
paper_authors.user: ksaenko
paper_title: COLA: How to adapt vision-language models to Compose Objects Localized
    with Attributes?
paper_summary: Compositional reasoning is a hallmark of human visual intelligence; yet
  despite the size of large vision-language models, they struggle to represent
  simple compositions by combining objects with their attributes. To measure this
  lack of compositional capability, we design Cola, a text-to-image retrieval
  benchmark to Compose Objects Localized with Attributes. Using Cola as a
  testbed, we explore modeling designs to adapt pre-trained vision-language
  models to reason compositionally about multiple attributes attached to multiple
  objects. We explore 6 finetuning strategies on 2 seminal vision-language
  models, using 3 finetuning datasets and 2 test benchmarks (Cola and CREPE).
  Surprisingly, our optimal finetuning strategy improves a 151M parameter CLIP,
  which disjointly encodes image and language during pretraining, to perform as
  well as a 241M parameter FLAVA, which uses a multi-modal transformer encoder
  during pretraining to attend over both vision and language modalities. This
  optimal finetuning strategy is a lightweight multi-modal adapter that jointly
  attends over both image and language features generated by the pretrained
  model. We show this works better than common strategies such as
  prompt/fine-tuning, or tuning a comparable number of unimodal layers.
paper_ai_keywords: compositional reasoning vision-language models Cola text-to-image retrieval Cola benchmark Compose Objects Localized with Attributes finetuning strategies pre-trained vision-language models multi-modal transformer encoder CLIP FLAVA multi-modal adapter prompt/fine-tuning unimodal layers