Retrieval-Enhanced Contrastive Vision-Text Models
Contrastive image-text models such as CLIP form the building blocks of many
state-of-the-art systems. While they excel at recognizing common generic
concepts, they still struggle on fine-grained entities which are rare, or even
absent from the pre-training dataset. Hence, a key ingredient to their success
has been the use of large-scale curated pre-training data aiming at expanding
the set of concepts that they can memorize during the pre-training stage. In
this work, we explore an alternative to encoding fine-grained knowledge
directly into the model's parameters: we instead train the model to retrieve
this knowledge from an external memory. Specifically, we propose to equip
existing vision-text models with the ability to refine their embedding with
cross-modal retrieved information from a memory at inference time, which
greatly improves their zero-shot predictions. Remarkably, we show that this can
be done with a light-weight, single-layer, fusion transformer on top of a
frozen CLIP. Our experiments validate that our retrieval-enhanced contrastive
(RECO) training improves CLIP performance substantially on several challenging
fine-grained tasks: for example +10.9 on Stanford Cars, +10.2 on CUB-2011 and
+7.3 on the recent OVEN benchmark.
6487e5a0ec2b3e1f38121856
Ahmet Iscen
claimed_verified
2023-06-13T17:39:18.630Z
64885bd4dfa0998196b1cf0a
/avatars/c3e2bbe793cdd146aea27c0011aace71.svg
Ahmet Iscen
user
ahmetius
6487e5a0ec2b3e1f38121857
Mathilde Caron
admin_assigned
2023-06-14T12:31:29.032Z
63139a8be29fb2e86d5fe7a4
/avatars/8938ec33144e58e6f9b6def4d38179d8.svg
Mathilde Caron
user
MathildeC
6487e5a0ec2b3e1f38121858
Alireza Fathi
admin_assigned
2023-06-14T12:31:18.604Z
645685170a1ea2229a28dbb2
/avatars/b83e043784b6db225ae4d0422aabbb3a.svg
alireza fathi
user
alireza007
6487e5a0ec2b3e1f38121859
Cordelia Schmid
Retrieval-Enhanced Contrastive Vision-Text Models
Contrastive image-text models such as CLIP form the building blocks of many
state-of-the-art systems. While they excel at recognizing common generic
concepts, they still struggle on fine-grained entities which are rare, or even
absent from the pre-training dataset. Hence, a key ingredient to their success
has been the use of large-scale curated pre-training data aiming at expanding
the set of concepts that they can memorize during the pre-training stage. In
this work, we explore an alternative to encoding fine-grained knowledge
directly into the model's parameters: we instead train the model to retrieve
this knowledge from an external memory. Specifically, we propose to equip
existing vision-text models with the ability to refine their embedding with
cross-modal retrieved information from a memory at inference time, which
greatly improves their zero-shot predictions. Remarkably, we show that this can
be done with a light-weight, single-layer, fusion transformer on top of a
frozen CLIP. Our experiments validate that our retrieval-enhanced contrastive
(RECO) training improves CLIP performance substantially on several challenging
fine-grained tasks: for example +10.9 on Stanford Cars, +10.2 on CUB-2011 and
+7.3 on the recent OVEN benchmark.
RECO, a retrieval-enhanced technique using a lightweight fusion transformer, improves CLIP's performance on fine-grained tasks by refining embeddings with cross-modal information from external memory.
contrastive image-text models CLIP fine-grained knowledge pre-training data external memory cross-modal retrieval fusion transformer zero-shot predictions Stanford Cars CUB-2011 OVEN benchmark