title: GeneCIS: A Benchmark for General Conditional Image Similarity
summary: We argue that there are many notions of 'similarity' and that models, like
  humans, should be able to adapt to these dynamically. This contrasts with most
  representation learning methods, supervised or self-supervised, which learn a
  fixed embedding function and hence implicitly assume a single notion of
  similarity. For instance, models trained on ImageNet are biased towards object
  categories, while a user might prefer the model to focus on colors, textures or
  specific elements in the scene. In this paper, we propose the GeneCIS
  ('genesis') benchmark, which measures models' ability to adapt to a range of
  similarity conditions. Extending prior work, our benchmark is designed for
  zero-shot evaluation only, and hence considers an open-set of similarity
  conditions. We find that baselines from powerful CLIP models struggle on
  GeneCIS and that performance on the benchmark is only weakly correlated with
  ImageNet accuracy, suggesting that simply scaling existing methods is not
  fruitful. We further propose a simple, scalable solution based on automatically
  mining information from existing image-caption datasets. We find our method
  offers a substantial boost over the baselines on GeneCIS, and further improves
  zero-shot performance on related image retrieval benchmarks. In fact, though
  evaluated zero-shot, our model surpasses state-of-the-art supervised models on
  MIT-States. Project page at https://sgvaze.github.io/genecis/.
paper_authors._id: 648939664cf3474dd1041d51
paper_authors.name: Sagar Vaze
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-14T12:12:04.852Z
paper_authors._id: 643838e5c5a91b84ece168dd
paper_authors.avatarUrl: /avatars/abebd42399decafbccc8579faa34e7d3.svg
paper_authors.fullname: Sagar Vaze
paper_authors.type: user
paper_authors.user: sgvaze
paper_authors._id: 648939664cf3474dd1041d52
paper_authors.name: Nicolas Carion
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-14T12:12:21.063Z
paper_authors._id: 6253c57927a8414d3bbe6d50
paper_authors.avatarUrl: /avatars/9d7cb38db7831a05e35c157a71f5be4a.svg
paper_authors.fullname: Nicolas Carion
paper_authors.type: user
paper_authors.user: alcinos
paper_authors._id: 648939664cf3474dd1041d53
paper_authors.name: Ishan Misra
paper_title: GeneCIS: A Benchmark for General Conditional Image Similarity
paper_summary: We argue that there are many notions of 'similarity' and that models, like
  humans, should be able to adapt to these dynamically. This contrasts with most
  representation learning methods, supervised or self-supervised, which learn a
  fixed embedding function and hence implicitly assume a single notion of
  similarity. For instance, models trained on ImageNet are biased towards object
  categories, while a user might prefer the model to focus on colors, textures or
  specific elements in the scene. In this paper, we propose the GeneCIS
  ('genesis') benchmark, which measures models' ability to adapt to a range of
  similarity conditions. Extending prior work, our benchmark is designed for
  zero-shot evaluation only, and hence considers an open-set of similarity
  conditions. We find that baselines from powerful CLIP models struggle on
  GeneCIS and that performance on the benchmark is only weakly correlated with
  ImageNet accuracy, suggesting that simply scaling existing methods is not
  fruitful. We further propose a simple, scalable solution based on automatically
  mining information from existing image-caption datasets. We find our method
  offers a substantial boost over the baselines on GeneCIS, and further improves
  zero-shot performance on related image retrieval benchmarks. In fact, though
  evaluated zero-shot, our model surpasses state-of-the-art supervised models on
  MIT-States. Project page at https://sgvaze.github.io/genecis/.
paper_ai_keywords: GeneCIS CLIP models zero-shot evaluation open-set ImageNet image-caption datasets image retrieval benchmarks MIT-States