title: VLM2Vec: Training Vision-Language Models for Massive Multimodal
    Embedding Tasks
summary: Embedding models have been crucial in enabling various downstream tasks such
  as semantic similarity, information retrieval, and clustering. Recently, there
  has been a surge of interest in developing universal text embedding models that
  can generalize across tasks (e.g., MTEB). However, progress in learning
  universal multimodal embedding models has been relatively slow despite their
  importance. In this work, we aim to explore the potential for building
  universal embeddings capable of handling a wide range of downstream tasks. Our
  contributions are twofold: (1) MMEB (Massive Multimodal Embedding Benchmark),
  which covers 4 meta-tasks (i.e. classification, visual question answering,
  multimodal retrieval, and visual grounding) and 36 datasets, including 20
  training and 16 evaluation datasets, and (2) VLM2Vec (Vision-Language Model ->
  Vector), a contrastive training framework that converts any state-of-the-art
  vision-language model into an embedding model via training on MMEB. Unlike
  previous models such as CLIP and BLIP, VLM2Vec can process any combination of
  images and text to generate a fixed-dimensional vector based on task
  instructions. We build a series of VLM2Vec models on Phi-3.5-V and evaluate
  them on MMEB's evaluation split. Our results show that \model achieves an
  absolute average improvement of 10% to 20% over existing multimodal embedding
  models on both in-distribution and out-of-distribution datasets in MMEB.
paper_authors._id: 6708056d0890f205c2aa0353
paper_authors.name: Ziyan Jiang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-10T16:48:47.379Z
paper_authors._id: 64778fb8168cb428e00f69b0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64778fb8168cb428e00f69b0/D_XYg74zHG9K3HUJj_gD4.jpeg
paper_authors.fullname: Ziyan Jiang
paper_authors.type: user
paper_authors.user: ziyjiang
paper_authors._id: 6708056d0890f205c2aa0354
paper_authors.name: Rui Meng
paper_authors._id: 6708056d0890f205c2aa0355
paper_authors.name: Xinyi Yang
paper_authors._id: 6708056d0890f205c2aa0356
paper_authors.name: Semih Yavuz
paper_authors._id: 6708056d0890f205c2aa0357
paper_authors.name: Yingbo Zhou
paper_authors._id: 6708056d0890f205c2aa0358
paper_authors.name: Wenhu Chen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-10T16:48:47.379Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_title: VLM2Vec: Training Vision-Language Models for Massive Multimodal
    Embedding Tasks
paper_summary: Embedding models have been crucial in enabling various downstream tasks such
  as semantic similarity, information retrieval, and clustering. Recently, there
  has been a surge of interest in developing universal text embedding models that
  can generalize across tasks (e.g., MTEB). However, progress in learning
  universal multimodal embedding models has been relatively slow despite their
  importance. In this work, we aim to explore the potential for building
  universal embeddings capable of handling a wide range of downstream tasks. Our
  contributions are twofold: (1) MMEB (Massive Multimodal Embedding Benchmark),
  which covers 4 meta-tasks (i.e. classification, visual question answering,
  multimodal retrieval, and visual grounding) and 36 datasets, including 20
  training and 16 evaluation datasets, and (2) VLM2Vec (Vision-Language Model ->
  Vector), a contrastive training framework that converts any state-of-the-art
  vision-language model into an embedding model via training on MMEB. Unlike
  previous models such as CLIP and BLIP, VLM2Vec can process any combination of
  images and text to generate a fixed-dimensional vector based on task
  instructions. We build a series of VLM2Vec models on Phi-3.5-V and evaluate
  them on MMEB's evaluation split. Our results show that \model achieves an
  absolute average improvement of 10% to 20% over existing multimodal embedding
  models on both in-distribution and out-of-distribution datasets in MMEB.
paper_ai_keywords: embedding models semantic similarity information retrieval clustering universal text embedding models MTEB universal multimodal embedding models MMEB (Massive Multimodal Embedding Benchmark) classification visual question answering multimodal retrieval visual grounding VLM2Vec (Vision-Language Model -> Vector) contrastive training framework vision-language model fixed-dimensional vector task instructions in-distribution out-of-distribution datasets