title: ABC: Achieving Better Control of Multimodal Embeddings using VLMs
summary: Visual embedding models excel at zero-shot tasks like visual retrieval and
  classification. However, these models cannot be used for tasks that contain
  ambiguity or require user instruction. These tasks necessitate a multimodal
  embedding model, which outputs embeddings that combine visual and natural
  language input. Existing CLIP-based approaches embed images and text
  independently, and fuse the result. We find that this results in weak
  interactions between modalities, and poor user control over the representation.
  We introduce ABC, an open-source multimodal embedding model that uses a
  vision-language model backbone to deeply integrate image features with natural
  language instructions. ABC achieves bestfor-size performance on MSCOCO
  image-to-text retrieval and is the top performing model on classification and
  VQA tasks in the Massive Multimodal Embedding Benchmark. With a strongly
  unified vision-language representation, ABC can use natural language to solve
  subtle and potentially ambiguous visual retrieval problems. To evaluate this
  capability, we design CtrlBench, a benchmark that requires interleaving textual
  instructions with image content for correct retrieval. ABC advances the state
  of multimodal embeddings by offering high-quality representations and flexible
  natural language control. Our model and datasets are available at our project
  page.
paper_authors._id: 67c755f898a2e37274c62c96
paper_authors.name: Benjamin Schneider
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-06T19:50:52.715Z
paper_authors._id: 6669ffeecf27eed0f2f2581e
paper_authors.avatarUrl: /avatars/6a1012cce15da12fa3bd6f9e0f87d100.svg
paper_authors.fullname: Benjamin Schneider
paper_authors.type: user
paper_authors.user: BenSchneider
paper_authors._id: 67c755f898a2e37274c62c97
paper_authors.name: Florian Kerschbaum
paper_authors._id: 67c755f898a2e37274c62c98
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-06T09:50:07.881Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_title: ABC: Achieving Better Control of Multimodal Embeddings using VLMs
paper_summary: Visual embedding models excel at zero-shot tasks like visual retrieval and
  classification. However, these models cannot be used for tasks that contain
  ambiguity or require user instruction. These tasks necessitate a multimodal
  embedding model, which outputs embeddings that combine visual and natural
  language input. Existing CLIP-based approaches embed images and text
  independently, and fuse the result. We find that this results in weak
  interactions between modalities, and poor user control over the representation.
  We introduce ABC, an open-source multimodal embedding model that uses a
  vision-language model backbone to deeply integrate image features with natural
  language instructions. ABC achieves bestfor-size performance on MSCOCO
  image-to-text retrieval and is the top performing model on classification and
  VQA tasks in the Massive Multimodal Embedding Benchmark. With a strongly
  unified vision-language representation, ABC can use natural language to solve
  subtle and potentially ambiguous visual retrieval problems. To evaluate this
  capability, we design CtrlBench, a benchmark that requires interleaving textual
  instructions with image content for correct retrieval. ABC advances the state
  of multimodal embeddings by offering high-quality representations and flexible
  natural language control. Our model and datasets are available at our project
  page.
paper_ai_summary: ABC is a multimodal embedding model using a vision-language backbone that achieves top performance on MSCOCO, classification, and VQA tasks, offering high-quality representations with flexible natural language control.
paper_ai_keywords: visual embedding models zero-shot tasks multimodal embedding model CLIP-based approaches image features natural language instructions MSCOCO image-to-text retrieval classification VQA tasks Massive Multimodal Embedding Benchmark visual retrieval problems CtrlBench