title: Modality Curation: Building Universal Embeddings for Advanced Multimodal
    Information Retrieval
summary: Multimodal information retrieval (MIR) faces inherent challenges due to the
  heterogeneity of data sources and the complexity of cross-modal alignment.
  While previous studies have identified modal gaps in feature spaces, a
  systematic approach to address these challenges remains unexplored. In this
  work, we introduce UNITE, a universal framework that tackles these challenges
  through two critical yet underexplored aspects: data curation and
  modality-aware training configurations. Our work provides the first
  comprehensive analysis of how modality-specific data properties influence
  downstream task performance across diverse scenarios. Moreover, we propose
  Modal-Aware Masked Contrastive Learning (MAMCL) to mitigate the competitive
  relationships among the instances of different modalities. Our framework
  achieves state-of-the-art results on multiple multimodal retrieval benchmarks,
  outperforming existing methods by notable margins. Through extensive
  experiments, we demonstrate that strategic modality curation and tailored
  training protocols are pivotal for robust cross-modal representation learning.
  This work not only advances MIR performance but also provides a foundational
  blueprint for future research in multimodal systems. Our project is available
  at https://friedrichor.github.io/projects/UNITE.
paper_authors._id: 6836c8391314d4ac39aebebb
paper_authors.name: Fanheng Kong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T08:41:20.041Z
paper_authors._id: 63835dc85c83390fc7527849
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63835dc85c83390fc7527849/axIViCepzduN3IfXWmj5A.png
paper_authors.fullname: Kong
paper_authors.type: user
paper_authors.user: friedrichor
paper_authors._id: 6836c8391314d4ac39aebebc
paper_authors.name: Jingyuan Zhang
paper_authors._id: 6836c8391314d4ac39aebebd
paper_authors.name: Yahui Liu
paper_authors._id: 6836c8391314d4ac39aebebe
paper_authors.name: Hongzhi Zhang
paper_authors._id: 6836c8391314d4ac39aebebf
paper_authors.name: Shi Feng
paper_authors._id: 6836c8391314d4ac39aebec0
paper_authors.name: Xiaocui Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:29:53.979Z
paper_authors._id: 649596a46d1cf21b50ed0ba7
paper_authors.avatarUrl: /avatars/47fec77d7067a2fc5fa8088a3b9af0f5.svg
paper_authors.fullname: Yang Xiaocui
paper_authors.type: user
paper_authors.user: YangXiaocui
paper_authors._id: 6836c8391314d4ac39aebec1
paper_authors.name: Daling Wang
paper_authors._id: 6836c8391314d4ac39aebec2
paper_authors.name: Yu Tian
paper_authors._id: 6836c8391314d4ac39aebec3
paper_authors.name: Victoria W.
paper_authors._id: 6836c8391314d4ac39aebec4
paper_authors.name: Fuzheng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:30:12.107Z
paper_authors._id: 67c5945da1661d5fa6f29adb
paper_authors.avatarUrl: /avatars/62561f3875c0c251cae949acc38d72dc.svg
paper_authors.fullname: Fuzheng Zhang
paper_authors.type: user
paper_authors.user: Edrex
paper_authors._id: 6836c8391314d4ac39aebec5
paper_authors.name: Guorui Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:30:18.800Z
paper_authors._id: 67c6c570cf87e2d2ebfc81aa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/67c6c570cf87e2d2ebfc81aa/7qAstZtIT86Uwrz3u_anv.jpeg
paper_authors.fullname: Guorui Zhou
paper_authors.type: user
paper_authors.user: GuoruiZhou
paper_title: Modality Curation: Building Universal Embeddings for Advanced Multimodal
    Information Retrieval
paper_summary: Multimodal information retrieval (MIR) faces inherent challenges due to the
  heterogeneity of data sources and the complexity of cross-modal alignment.
  While previous studies have identified modal gaps in feature spaces, a
  systematic approach to address these challenges remains unexplored. In this
  work, we introduce UNITE, a universal framework that tackles these challenges
  through two critical yet underexplored aspects: data curation and
  modality-aware training configurations. Our work provides the first
  comprehensive analysis of how modality-specific data properties influence
  downstream task performance across diverse scenarios. Moreover, we propose
  Modal-Aware Masked Contrastive Learning (MAMCL) to mitigate the competitive
  relationships among the instances of different modalities. Our framework
  achieves state-of-the-art results on multiple multimodal retrieval benchmarks,
  outperforming existing methods by notable margins. Through extensive
  experiments, we demonstrate that strategic modality curation and tailored
  training protocols are pivotal for robust cross-modal representation learning.
  This work not only advances MIR performance but also provides a foundational
  blueprint for future research in multimodal systems. Our project is available
  at https://friedrichor.github.io/projects/UNITE.
paper_ai_summary: UNITE addresses challenges in multimodal information retrieval through data curation and modality-aware training, achieving state-of-the-art results across benchmarks with Modal-Aware Masked Contrastive Learning.
paper_ai_keywords: MAMCL Modal-Aware Masked Contrastive Learning multimodal information retrieval modality-specific data properties cross-modal alignment cross-modal representation learning