VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and
  Visual Documents
Multimodal embedding models have been crucial in enabling various downstream
tasks such as semantic similarity, information retrieval, and clustering over
different modalities. However, existing multimodal embeddings like VLM2Vec,
E5-V, GME are predominantly focused on natural images, with limited support for
other visual forms such as videos and visual documents. This restricts their
applicability in real-world scenarios, including AI agents, multi-modal search
and recommendation, and retrieval-augmented generation (RAG). To close this
gap, we propose VLM2Vec-V2, a unified framework for learning embeddings across
diverse visual forms. First, we introduce MMEB-V2, a comprehensive benchmark
that extends MMEB with five new task types: visual document retrieval, video
retrieval, temporal grounding, video classification and video question
answering - spanning text, image, video, and visual document inputs. Next, we
train VLM2Vec-V2, a general-purpose embedding model that supports text, image,
video, and visual document inputs. Extensive experiments show that VLM2Vec-V2
achieves strong performance not only on the newly introduced video and document
retrieval tasks, but also improves over prior baselines on the original image
benchmarks. Through extensive evaluation, our study offers insights into the
generalizability of various multimodal embedding models and highlights
effective strategies for unified embedding learning, laying the groundwork for
more scalable and adaptable representation learning in both research and
real-world settings.
686c96a0364e2ad167eb540c
Rui Meng
686c96a0364e2ad167eb540d
Ziyan Jiang
claimed_verified
2025-07-08T08:06:15.117Z
64778fb8168cb428e00f69b0
https://cdn-avatars.huggingface.co/v1/production/uploads/64778fb8168cb428e00f69b0/D_XYg74zHG9K3HUJj_gD4.jpeg
Ziyan Jiang
user
ziyjiang
686c96a0364e2ad167eb540e
Ye Liu
686c96a0364e2ad167eb540f
Mingyi Su
claimed_verified
2025-07-11T08:08:49.077Z
6721742a6978ab6a7713ee37
https://cdn-avatars.huggingface.co/v1/production/uploads/6721742a6978ab6a7713ee37/6VZN9ERY4NS4yUOFDGy3I.jpeg
MINGYI SU
user
MINGYISU
686c96a0364e2ad167eb5410
Xinyi Yang
686c96a0364e2ad167eb5411
Yuepeng Fu
686c96a0364e2ad167eb5412
Can Qin
686c96a0364e2ad167eb5413
Zeyuan Chen
686c96a0364e2ad167eb5414
Ran Xu
686c96a0364e2ad167eb5415
Caiming Xiong
686c96a0364e2ad167eb5416
Yingbo Zhou
686c96a0364e2ad167eb5417
Wenhu Chen
686c96a0364e2ad167eb5418
Semih Yavuz
VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and
  Visual Documents
Multimodal embedding models have been crucial in enabling various downstream
tasks such as semantic similarity, information retrieval, and clustering over
different modalities. However, existing multimodal embeddings like VLM2Vec,
E5-V, GME are predominantly focused on natural images, with limited support for
other visual forms such as videos and visual documents. This restricts their
applicability in real-world scenarios, including AI agents, multi-modal search
and recommendation, and retrieval-augmented generation (RAG). To close this
gap, we propose VLM2Vec-V2, a unified framework for learning embeddings across
diverse visual forms. First, we introduce MMEB-V2, a comprehensive benchmark
that extends MMEB with five new task types: visual document retrieval, video
retrieval, temporal grounding, video classification and video question
answering - spanning text, image, video, and visual document inputs. Next, we
train VLM2Vec-V2, a general-purpose embedding model that supports text, image,
video, and visual document inputs. Extensive experiments show that VLM2Vec-V2
achieves strong performance not only on the newly introduced video and document
retrieval tasks, but also improves over prior baselines on the original image
benchmarks. Through extensive evaluation, our study offers insights into the
generalizability of various multimodal embedding models and highlights
effective strategies for unified embedding learning, laying the groundwork for
more scalable and adaptable representation learning in both research and
real-world settings.
A unified framework VLM2Vec-V2 for multimodal embedding supports diverse visual forms, including videos and visual documents, improving performance across various tasks and benchmarks.
MMEB-V2 VLM2Vec-V2 visual document retrieval video retrieval temporal grounding video classification video question answering multimodal embedding models unified embedding learning