title: mmE5: Improving Multimodal Multilingual Embeddings via High-quality
    Synthetic Data
summary: Multimodal embedding models have gained significant attention for their
  ability to map data from different modalities, such as text and images, into a
  unified representation space. However, the limited labeled multimodal data
  often hinders embedding performance. Recent approaches have leveraged data
  synthesis to address this problem, yet the quality of synthetic data remains a
  critical bottleneck. In this work, we identify three criteria for high-quality
  synthetic multimodal data. First, broad scope ensures that the generated data
  covers diverse tasks and modalities, making it applicable to various downstream
  scenarios. Second, robust cross-modal alignment makes different modalities
  semantically consistent. Third, high fidelity ensures that the synthetic data
  maintains realistic details to enhance its reliability. Guided by these
  principles, we synthesize datasets that: (1) cover a wide range of tasks,
  modality combinations, and languages, (2) are generated via a deep thinking
  process within a single pass of a multimodal large language model, and (3)
  incorporate real-world images with accurate and relevant texts, ensuring
  fidelity through self-evaluation and refinement. Leveraging these high-quality
  synthetic and labeled datasets, we train a multimodal multilingual E5 model
  mmE5. Extensive experiments demonstrate that mmE5 achieves state-of-the-art
  performance on the MMEB Benchmark and superior multilingual performance on the
  XTD benchmark. Our codes, datasets and models are released in
  https://github.com/haon-chen/mmE5.
paper_authors._id: 67ad5f3fcad644864b4366ca
paper_authors.name: Haonan Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T08:21:55.329Z
paper_authors._id: 66add675c7a575aa0e03d5f3
paper_authors.avatarUrl: /avatars/b72b18130664c1de197c1f8df371aa70.svg
paper_authors.fullname: Haonan Chen
paper_authors.type: user
paper_authors.user: Haon-Chen
paper_authors._id: 67ad5f3fcad644864b4366cb
paper_authors.name: Liang Wang
paper_authors._id: 67ad5f3fcad644864b4366cc
paper_authors.name: Nan Yang
paper_authors._id: 67ad5f3fcad644864b4366cd
paper_authors.name: Yutao Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:53:09.223Z
paper_authors._id: 625e62452a7279d3c77b5c38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/625e62452a7279d3c77b5c38/zJINew6U4_Gup4WTobb-0.jpeg
paper_authors.fullname: Yutao Zhu
paper_authors.type: user
paper_authors.user: yutaozhu94
paper_authors._id: 67ad5f3fcad644864b4366ce
paper_authors.name: Ziliang Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:53:01.987Z
paper_authors._id: 6639d5c106b25a7ea6f18391
paper_authors.avatarUrl: /avatars/788e339472999a9159f77f857817d618.svg
paper_authors.fullname: Ziliang Zhao
paper_authors.type: user
paper_authors.user: ZillionZhao
paper_authors._id: 67ad5f3fcad644864b4366cf
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:52:40.042Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_authors._id: 67ad5f3fcad644864b4366d0
paper_authors.name: Zhicheng Dou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T12:52:33.880Z
paper_authors._id: 66f0bf59e9d50ec57febf751
paper_authors.avatarUrl: /avatars/be97941e60064e5dd806c6fe9db3c537.svg
paper_authors.fullname: Zhicheng Dou
paper_authors.type: user
paper_authors.user: douzc
paper_title: mmE5: Improving Multimodal Multilingual Embeddings via High-quality
    Synthetic Data
paper_summary: Multimodal embedding models have gained significant attention for their
  ability to map data from different modalities, such as text and images, into a
  unified representation space. However, the limited labeled multimodal data
  often hinders embedding performance. Recent approaches have leveraged data
  synthesis to address this problem, yet the quality of synthetic data remains a
  critical bottleneck. In this work, we identify three criteria for high-quality
  synthetic multimodal data. First, broad scope ensures that the generated data
  covers diverse tasks and modalities, making it applicable to various downstream
  scenarios. Second, robust cross-modal alignment makes different modalities
  semantically consistent. Third, high fidelity ensures that the synthetic data
  maintains realistic details to enhance its reliability. Guided by these
  principles, we synthesize datasets that: (1) cover a wide range of tasks,
  modality combinations, and languages, (2) are generated via a deep thinking
  process within a single pass of a multimodal large language model, and (3)
  incorporate real-world images with accurate and relevant texts, ensuring
  fidelity through self-evaluation and refinement. Leveraging these high-quality
  synthetic and labeled datasets, we train a multimodal multilingual E5 model
  mmE5. Extensive experiments demonstrate that mmE5 achieves state-of-the-art
  performance on the MMEB Benchmark and superior multilingual performance on the
  XTD benchmark. Our codes, datasets and models are released in
  https://github.com/haon-chen/mmE5.
paper_ai_summary: A deep learning approach synthesizes high-quality multimodal data, enhancing mmE5 model performance on benchmark tasks in multilingual and cross-modal embeddings.
paper_ai_keywords: multimodal embedding models labeled multimodal data data synthesis deep thinking process multimodal large language model MMEB Benchmark XTD benchmark