COLA: How to adapt vision-language models to Compose Objects Localized
  with Attributes?
Compositional reasoning is a hallmark of human visual intelligence; yet
despite the size of large vision-language models, they struggle to represent
simple compositions by combining objects with their attributes. To measure this
lack of compositional capability, we design Cola, a text-to-image retrieval
benchmark to Compose Objects Localized with Attributes. Using Cola as a
testbed, we explore modeling designs to adapt pre-trained vision-language
models to reason compositionally about multiple attributes attached to multiple
objects. We explore 6 finetuning strategies on 2 seminal vision-language
models, using 3 finetuning datasets and 2 test benchmarks (Cola and CREPE).
Surprisingly, our optimal finetuning strategy improves a 151M parameter CLIP,
which disjointly encodes image and language during pretraining, to perform as
well as a 241M parameter FLAVA, which uses a multi-modal transformer encoder
during pretraining to attend over both vision and language modalities. This
optimal finetuning strategy is a lightweight multi-modal adapter that jointly
attends over both image and language features generated by the pretrained
model. We show this works better than common strategies such as
prompt/fine-tuning, or tuning a comparable number of unimodal layers.
64585a86aa426bae5e07231a
Arijit Ray
claimed_verified
2023-06-08T16:08:50.617Z
6303f2f01dd5d3c6248559ae
https://cdn-avatars.huggingface.co/v1/production/uploads/6303f2f01dd5d3c6248559ae/1l18-bKuTcWSRZdukBFDm.jpeg
Arijit
user
array
64585a86aa426bae5e07231b
Filip Radenovic
64585a86aa426bae5e07231c
Abhimanyu Dubey
64585a86aa426bae5e07231d
Bryan A. Plummer
64585a86aa426bae5e07231e
Ranjay Krishna
64585a86aa426bae5e07231f
Kate Saenko
claimed_verified
2023-06-05T10:07:05.146Z
64414f17ad24e9b2cfbd5c29
/avatars/9e527ef654424d96e5c29d9f76979341.svg
K S
user
ksaenko
COLA: How to adapt vision-language models to Compose Objects Localized
  with Attributes?
Compositional reasoning is a hallmark of human visual intelligence; yet
despite the size of large vision-language models, they struggle to represent
simple compositions by combining objects with their attributes. To measure this
lack of compositional capability, we design Cola, a text-to-image retrieval
benchmark to Compose Objects Localized with Attributes. Using Cola as a
testbed, we explore modeling designs to adapt pre-trained vision-language
models to reason compositionally about multiple attributes attached to multiple
objects. We explore 6 finetuning strategies on 2 seminal vision-language
models, using 3 finetuning datasets and 2 test benchmarks (Cola and CREPE).
Surprisingly, our optimal finetuning strategy improves a 151M parameter CLIP,
which disjointly encodes image and language during pretraining, to perform as
well as a 241M parameter FLAVA, which uses a multi-modal transformer encoder
during pretraining to attend over both vision and language modalities. This
optimal finetuning strategy is a lightweight multi-modal adapter that jointly
attends over both image and language features generated by the pretrained
model. We show this works better than common strategies such as
prompt/fine-tuning, or tuning a comparable number of unimodal layers.
A novel text-to-image retrieval benchmark (Cola) is used to assess and enhance compositional reasoning in vision-language models through fine-tuning strategies.
vision-language models compositional reasoning text-to-image retrieval Cola multi-modal transformer pretraining fine-tuning strategies CLIP FLAVA multi-modal adapter