title: Breaking the Modality Barrier: Universal Embedding Learning with
    Multimodal LLMs
summary: The Contrastive Language-Image Pre-training (CLIP) framework has become a
  widely used approach for multimodal representation learning, particularly in
  image-text retrieval and clustering. However, its efficacy is constrained by
  three key limitations: (1) text token truncation, (2) isolated image-text
  encoding, and (3) deficient compositionality due to bag-of-words behavior.
  While recent Multimodal Large Language Models (MLLMs) have demonstrated
  significant advances in generalized vision-language understanding, their
  potential for learning transferable multimodal representations remains
  underexplored.In this work, we present UniME (Universal Multimodal Embedding),
  a novel two-stage framework that leverages MLLMs to learn discriminative
  representations for diverse downstream tasks. In the first stage, we perform
  textual discriminative knowledge distillation from a powerful LLM-based teacher
  model to enhance the embedding capability of the MLLM\'s language component. In
  the second stage, we introduce hard negative enhanced instruction tuning to
  further advance discriminative representation learning. Specifically, we
  initially mitigate false negative contamination and then sample multiple hard
  negatives per instance within each batch, forcing the model to focus on
  challenging samples. This approach not only improves discriminative power but
  also enhances instruction-following ability in downstream tasks. We conduct
  extensive experiments on the MMEB benchmark and multiple retrieval tasks,
  including short and long caption retrieval and compositional retrieval. Results
  demonstrate that UniME achieves consistent performance improvement across all
  tasks, exhibiting superior discriminative and compositional capabilities.
paper_authors._id: 680adfbe464a44cea0b843c1
paper_authors.name: Tiancheng Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-26T08:53:46.072Z
paper_authors._id: 641030c77a15af878ae5bd8f
paper_authors.avatarUrl: /avatars/8a5037edf55c78ebc317c8b191343671.svg
paper_authors.fullname: TianchengGu
paper_authors.type: user
paper_authors.user: TianchengGu
paper_authors._id: 680adfbe464a44cea0b843c2
paper_authors.name: Kaicheng Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-25T08:34:46.935Z
paper_authors._id: 63e202f352b7578dba448ab5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e202f352b7578dba448ab5/8itVBLcv14m7OVsoF8h1o.jpeg
paper_authors.fullname: Yang
paper_authors.type: user
paper_authors.user: Kaichengalex
paper_authors._id: 680adfbe464a44cea0b843c3
paper_authors.name: Ziyong Feng
paper_authors._id: 680adfbe464a44cea0b843c4
paper_authors.name: Xingjun Wang
paper_authors._id: 680adfbe464a44cea0b843c5
paper_authors.name: Yanzhao Zhang
paper_authors._id: 680adfbe464a44cea0b843c6
paper_authors.name: Dingkun Long
paper_authors._id: 680adfbe464a44cea0b843c7
paper_authors.name: Yingda Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-30T07:56:53.763Z
paper_authors._id: 6215ca5692c0ecfba9186921
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1645595695434-6215ca5692c0ecfba9186921.png
paper_authors.fullname: Yingda Chen
paper_authors.type: user
paper_authors.user: Yingda
paper_authors._id: 680adfbe464a44cea0b843c8
paper_authors.name: Weidong Cai
paper_authors._id: 680adfbe464a44cea0b843c9
paper_authors.name: Jiankang Deng
paper_title: Breaking the Modality Barrier: Universal Embedding Learning with
    Multimodal LLMs
paper_summary: The Contrastive Language-Image Pre-training (CLIP) framework has become a
  widely used approach for multimodal representation learning, particularly in
  image-text retrieval and clustering. However, its efficacy is constrained by
  three key limitations: (1) text token truncation, (2) isolated image-text
  encoding, and (3) deficient compositionality due to bag-of-words behavior.
  While recent Multimodal Large Language Models (MLLMs) have demonstrated
  significant advances in generalized vision-language understanding, their
  potential for learning transferable multimodal representations remains
  underexplored.In this work, we present UniME (Universal Multimodal Embedding),
  a novel two-stage framework that leverages MLLMs to learn discriminative
  representations for diverse downstream tasks. In the first stage, we perform
  textual discriminative knowledge distillation from a powerful LLM-based teacher
  model to enhance the embedding capability of the MLLM\'s language component. In
  the second stage, we introduce hard negative enhanced instruction tuning to
  further advance discriminative representation learning. Specifically, we
  initially mitigate false negative contamination and then sample multiple hard
  negatives per instance within each batch, forcing the model to focus on
  challenging samples. This approach not only improves discriminative power but
  also enhances instruction-following ability in downstream tasks. We conduct
  extensive experiments on the MMEB benchmark and multiple retrieval tasks,
  including short and long caption retrieval and compositional retrieval. Results
  demonstrate that UniME achieves consistent performance improvement across all
  tasks, exhibiting superior discriminative and compositional capabilities.
paper_ai_summary: A two-stage framework using Multimodal Large Language Models learns discriminative embeddings for various tasks by distilling knowledge from a powerful LLM and enhancing instruction tuning with hard negatives.
paper_ai_keywords: Contrastive Language-Image Pre-training CLIP multimodal representation learning image-text retrieval Multimodal Large Language Models MLLMs Universal Multimodal Embedding UniME textual discriminative knowledge distillation hard negative enhanced instruction tuning discriminative representation learning MMEB benchmark compositional retrieval