mmE5: Improving Multimodal Multilingual Embeddings via High-quality
  Synthetic Data
Multimodal embedding models have gained significant attention for their
ability to map data from different modalities, such as text and images, into a
unified representation space. However, the limited labeled multimodal data
often hinders embedding performance. Recent approaches have leveraged data
synthesis to address this problem, yet the quality of synthetic data remains a
critical bottleneck. In this work, we identify three criteria for high-quality
synthetic multimodal data. First, broad scope ensures that the generated data
covers diverse tasks and modalities, making it applicable to various downstream
scenarios. Second, robust cross-modal alignment makes different modalities
semantically consistent. Third, high fidelity ensures that the synthetic data
maintains realistic details to enhance its reliability. Guided by these
principles, we synthesize datasets that: (1) cover a wide range of tasks,
modality combinations, and languages, (2) are generated via a deep thinking
process within a single pass of a multimodal large language model, and (3)
incorporate real-world images with accurate and relevant texts, ensuring
fidelity through self-evaluation and refinement. Leveraging these high-quality
synthetic and labeled datasets, we train a multimodal multilingual E5 model
mmE5. Extensive experiments demonstrate that mmE5 achieves state-of-the-art
performance on the MMEB Benchmark and superior multilingual performance on the
XTD benchmark. Our codes, datasets and models are released in
https://github.com/haon-chen/mmE5.
67ad5f3fcad644864b4366ca
Haonan Chen
claimed_verified
2025-02-13T08:21:55.329Z
66add675c7a575aa0e03d5f3
/avatars/b72b18130664c1de197c1f8df371aa70.svg
Haonan Chen
user
Haon-Chen
67ad5f3fcad644864b4366cb
Liang Wang
67ad5f3fcad644864b4366cc
Nan Yang
67ad5f3fcad644864b4366cd
Yutao Zhu
admin_assigned
2025-02-14T12:53:09.223Z
625e62452a7279d3c77b5c38
https://cdn-avatars.huggingface.co/v1/production/uploads/625e62452a7279d3c77b5c38/zJINew6U4_Gup4WTobb-0.jpeg
Yutao Zhu
user
yutaozhu94
67ad5f3fcad644864b4366ce
Ziliang Zhao
admin_assigned
2025-02-14T12:53:01.987Z
6639d5c106b25a7ea6f18391
/avatars/788e339472999a9159f77f857817d618.svg
Ziliang Zhao
user
ZillionZhao
67ad5f3fcad644864b4366cf
Furu Wei
admin_assigned
2025-02-14T12:52:40.042Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
67ad5f3fcad644864b4366d0
Zhicheng Dou
admin_assigned
2025-02-14T12:52:33.880Z
66f0bf59e9d50ec57febf751
/avatars/be97941e60064e5dd806c6fe9db3c537.svg
Zhicheng Dou
user
douzc
mmE5: Improving Multimodal Multilingual Embeddings via High-quality
  Synthetic Data
Multimodal embedding models have gained significant attention for their
ability to map data from different modalities, such as text and images, into a
unified representation space. However, the limited labeled multimodal data
often hinders embedding performance. Recent approaches have leveraged data
synthesis to address this problem, yet the quality of synthetic data remains a
critical bottleneck. In this work, we identify three criteria for high-quality
synthetic multimodal data. First, broad scope ensures that the generated data
covers diverse tasks and modalities, making it applicable to various downstream
scenarios. Second, robust cross-modal alignment makes different modalities
semantically consistent. Third, high fidelity ensures that the synthetic data
maintains realistic details to enhance its reliability. Guided by these
principles, we synthesize datasets that: (1) cover a wide range of tasks,
modality combinations, and languages, (2) are generated via a deep thinking
process within a single pass of a multimodal large language model, and (3)
incorporate real-world images with accurate and relevant texts, ensuring
fidelity through self-evaluation and refinement. Leveraging these high-quality
synthetic and labeled datasets, we train a multimodal multilingual E5 model
mmE5. Extensive experiments demonstrate that mmE5 achieves state-of-the-art
performance on the MMEB Benchmark and superior multilingual performance on the
XTD benchmark. Our codes, datasets and models are released in
https://github.com/haon-chen/mmE5.
A deep learning approach synthesizes high-quality multimodal data, enhancing mmE5 model performance on benchmark tasks in multilingual and cross-modal embeddings.
multimodal embedding models labeled multimodal data data synthesis deep thinking process multimodal large language model MMEB Benchmark XTD benchmark