title: Retrieval-Enhanced Contrastive Vision-Text Models
summary: Contrastive image-text models such as CLIP form the building blocks of many
  state-of-the-art systems. While they excel at recognizing common generic
  concepts, they still struggle on fine-grained entities which are rare, or even
  absent from the pre-training dataset. Hence, a key ingredient to their success
  has been the use of large-scale curated pre-training data aiming at expanding
  the set of concepts that they can memorize during the pre-training stage. In
  this work, we explore an alternative to encoding fine-grained knowledge
  directly into the model's parameters: we instead train the model to retrieve
  this knowledge from an external memory. Specifically, we propose to equip
  existing vision-text models with the ability to refine their embedding with
  cross-modal retrieved information from a memory at inference time, which
  greatly improves their zero-shot predictions. Remarkably, we show that this can
  be done with a light-weight, single-layer, fusion transformer on top of a
  frozen CLIP. Our experiments validate that our retrieval-enhanced contrastive
  (RECO) training improves CLIP performance substantially on several challenging
  fine-grained tasks: for example +10.9 on Stanford Cars, +10.2 on CUB-2011 and
  +7.3 on the recent OVEN benchmark.
paper_authors._id: 6487e5a0ec2b3e1f38121856
paper_authors.name: Ahmet Iscen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-13T17:39:18.630Z
paper_authors._id: 64885bd4dfa0998196b1cf0a
paper_authors.avatarUrl: /avatars/c3e2bbe793cdd146aea27c0011aace71.svg
paper_authors.fullname: Ahmet Iscen
paper_authors.type: user
paper_authors.user: ahmetius
paper_authors._id: 6487e5a0ec2b3e1f38121857
paper_authors.name: Mathilde Caron
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-14T12:31:29.032Z
paper_authors._id: 63139a8be29fb2e86d5fe7a4
paper_authors.avatarUrl: /avatars/8938ec33144e58e6f9b6def4d38179d8.svg
paper_authors.fullname: Mathilde Caron
paper_authors.type: user
paper_authors.user: MathildeC
paper_authors._id: 6487e5a0ec2b3e1f38121858
paper_authors.name: Alireza Fathi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-14T12:31:18.604Z
paper_authors._id: 645685170a1ea2229a28dbb2
paper_authors.avatarUrl: /avatars/b83e043784b6db225ae4d0422aabbb3a.svg
paper_authors.fullname: alireza fathi
paper_authors.type: user
paper_authors.user: alireza007
paper_authors._id: 6487e5a0ec2b3e1f38121859
paper_authors.name: Cordelia Schmid
paper_title: Retrieval-Enhanced Contrastive Vision-Text Models
paper_summary: Contrastive image-text models such as CLIP form the building blocks of many
  state-of-the-art systems. While they excel at recognizing common generic
  concepts, they still struggle on fine-grained entities which are rare, or even
  absent from the pre-training dataset. Hence, a key ingredient to their success
  has been the use of large-scale curated pre-training data aiming at expanding
  the set of concepts that they can memorize during the pre-training stage. In
  this work, we explore an alternative to encoding fine-grained knowledge
  directly into the model's parameters: we instead train the model to retrieve
  this knowledge from an external memory. Specifically, we propose to equip
  existing vision-text models with the ability to refine their embedding with
  cross-modal retrieved information from a memory at inference time, which
  greatly improves their zero-shot predictions. Remarkably, we show that this can
  be done with a light-weight, single-layer, fusion transformer on top of a
  frozen CLIP. Our experiments validate that our retrieval-enhanced contrastive
  (RECO) training improves CLIP performance substantially on several challenging
  fine-grained tasks: for example +10.9 on Stanford Cars, +10.2 on CUB-2011 and
  +7.3 on the recent OVEN benchmark.
paper_ai_summary: RECO, a retrieval-enhanced technique using a lightweight fusion transformer, improves CLIP's performance on fine-grained tasks by refining embeddings with cross-modal information from external memory.
paper_ai_keywords: contrastive image-text models CLIP fine-grained knowledge pre-training data external memory cross-modal retrieval fusion transformer zero-shot predictions Stanford Cars CUB-2011 OVEN benchmark