Breaking the Modality Barrier: Universal Embedding Learning with
  Multimodal LLMs
The Contrastive Language-Image Pre-training (CLIP) framework has become a
widely used approach for multimodal representation learning, particularly in
image-text retrieval and clustering. However, its efficacy is constrained by
three key limitations: (1) text token truncation, (2) isolated image-text
encoding, and (3) deficient compositionality due to bag-of-words behavior.
While recent Multimodal Large Language Models (MLLMs) have demonstrated
significant advances in generalized vision-language understanding, their
potential for learning transferable multimodal representations remains
underexplored.In this work, we present UniME (Universal Multimodal Embedding),
a novel two-stage framework that leverages MLLMs to learn discriminative
representations for diverse downstream tasks. In the first stage, we perform
textual discriminative knowledge distillation from a powerful LLM-based teacher
model to enhance the embedding capability of the MLLM\'s language component. In
the second stage, we introduce hard negative enhanced instruction tuning to
further advance discriminative representation learning. Specifically, we
initially mitigate false negative contamination and then sample multiple hard
negatives per instance within each batch, forcing the model to focus on
challenging samples. This approach not only improves discriminative power but
also enhances instruction-following ability in downstream tasks. We conduct
extensive experiments on the MMEB benchmark and multiple retrieval tasks,
including short and long caption retrieval and compositional retrieval. Results
demonstrate that UniME achieves consistent performance improvement across all
tasks, exhibiting superior discriminative and compositional capabilities.
680adfbe464a44cea0b843c1
Tiancheng Gu
claimed_verified
2025-04-26T08:53:46.072Z
641030c77a15af878ae5bd8f
/avatars/8a5037edf55c78ebc317c8b191343671.svg
TianchengGu
user
TianchengGu
680adfbe464a44cea0b843c2
Kaicheng Yang
claimed_verified
2025-04-25T08:34:46.935Z
63e202f352b7578dba448ab5
https://cdn-avatars.huggingface.co/v1/production/uploads/63e202f352b7578dba448ab5/8itVBLcv14m7OVsoF8h1o.jpeg
Yang
user
Kaichengalex
680adfbe464a44cea0b843c3
Ziyong Feng
680adfbe464a44cea0b843c4
Xingjun Wang
claimed_verified
2025-07-17T08:20:11.785Z
62f612e22e53c2efd33943c6
https://cdn-avatars.huggingface.co/v1/production/uploads/1679249371815-62f612e22e53c2efd33943c6.jpeg
xingjun wang
user
wangxingjun778
680adfbe464a44cea0b843c5
Yanzhao Zhang
680adfbe464a44cea0b843c6
Dingkun Long
680adfbe464a44cea0b843c7
Yingda Chen
claimed_verified
2025-04-30T07:56:53.763Z
6215ca5692c0ecfba9186921
https://cdn-avatars.huggingface.co/v1/production/uploads/1645595695434-6215ca5692c0ecfba9186921.png
Yingda Chen
user
Yingda
680adfbe464a44cea0b843c8
Weidong Cai
680adfbe464a44cea0b843c9
Jiankang Deng
Breaking the Modality Barrier: Universal Embedding Learning with
  Multimodal LLMs
The Contrastive Language-Image Pre-training (CLIP) framework has become a
widely used approach for multimodal representation learning, particularly in
image-text retrieval and clustering. However, its efficacy is constrained by
three key limitations: (1) text token truncation, (2) isolated image-text
encoding, and (3) deficient compositionality due to bag-of-words behavior.
While recent Multimodal Large Language Models (MLLMs) have demonstrated
significant advances in generalized vision-language understanding, their
potential for learning transferable multimodal representations remains
underexplored.In this work, we present UniME (Universal Multimodal Embedding),
a novel two-stage framework that leverages MLLMs to learn discriminative
representations for diverse downstream tasks. In the first stage, we perform
textual discriminative knowledge distillation from a powerful LLM-based teacher
model to enhance the embedding capability of the MLLM\'s language component. In
the second stage, we introduce hard negative enhanced instruction tuning to
further advance discriminative representation learning. Specifically, we
initially mitigate false negative contamination and then sample multiple hard
negatives per instance within each batch, forcing the model to focus on
challenging samples. This approach not only improves discriminative power but
also enhances instruction-following ability in downstream tasks. We conduct
extensive experiments on the MMEB benchmark and multiple retrieval tasks,
including short and long caption retrieval and compositional retrieval. Results
demonstrate that UniME achieves consistent performance improvement across all
tasks, exhibiting superior discriminative and compositional capabilities.
A two-stage framework using Multimodal Large Language Models learns discriminative embeddings for various tasks by distilling knowledge from a powerful LLM and enhancing instruction tuning with hard negatives.
Contrastive Language-Image Pre-training CLIP multimodal representation learning image-text retrieval Multimodal Large Language Models MLLMs Universal Multimodal Embedding UniME textual discriminative knowledge distillation hard negative enhanced instruction tuning discriminative representation learning MMEB benchmark compositional retrieval