LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted
  Contrastive Learning
Universal multimodal embedding models play a critical role in tasks such as
interleaved image-text retrieval, multimodal RAG, and multimodal clustering.
However, our empirical results indicate that existing LMM-based embedding
models trained with the standard InfoNCE loss exhibit a high degree of overlap
in similarity distribution between positive and negative pairs, making it
challenging to distinguish hard negative pairs effectively. To deal with this
issue, we propose a simple yet effective framework that dynamically improves
the embedding model's representation learning for negative pairs based on their
discriminative difficulty. Within this framework, we train a series of models,
named LLaVE, and evaluate them on the MMEB benchmark, which covers 4 meta-tasks
and 36 datasets. Experimental results show that LLaVE establishes stronger
baselines that achieve state-of-the-art (SOTA) performance while demonstrating
strong scalability and efficiency. Specifically, LLaVE-2B surpasses the
previous SOTA 7B models, while LLaVE-7B achieves a further performance
improvement of 6.2 points. Although LLaVE is trained on image-text data, it can
generalize to text-video retrieval tasks in a zero-shot manner and achieve
strong performance, demonstrating its remarkable potential for transfer to
other embedding tasks.
67ce5542818e1825dea7440b
Zhibin Lan
claimed_verified
2025-03-10T08:00:54.535Z
6626449503e1f561573d30e9
/avatars/e7f9720ccd01bae32d0a03a1b0dacab5.svg
Zhibin Lan
user
zhibinlan
67ce5542818e1825dea7440c
Liqiang Niu
claimed_verified
2025-03-10T08:00:51.713Z
635239137d071f23d083b056
/avatars/1f1a0ed38d8de499d4b78922801c6d95.svg
liqiang niu
user
lqniu
67ce5542818e1825dea7440d
Fandong Meng
67ce5542818e1825dea7440e
Jie Zhou
67ce5542818e1825dea7440f
Jinsong Su
LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted
  Contrastive Learning
Universal multimodal embedding models play a critical role in tasks such as
interleaved image-text retrieval, multimodal RAG, and multimodal clustering.
However, our empirical results indicate that existing LMM-based embedding
models trained with the standard InfoNCE loss exhibit a high degree of overlap
in similarity distribution between positive and negative pairs, making it
challenging to distinguish hard negative pairs effectively. To deal with this
issue, we propose a simple yet effective framework that dynamically improves
the embedding model's representation learning for negative pairs based on their
discriminative difficulty. Within this framework, we train a series of models,
named LLaVE, and evaluate them on the MMEB benchmark, which covers 4 meta-tasks
and 36 datasets. Experimental results show that LLaVE establishes stronger
baselines that achieve state-of-the-art (SOTA) performance while demonstrating
strong scalability and efficiency. Specifically, LLaVE-2B surpasses the
previous SOTA 7B models, while LLaVE-7B achieves a further performance
improvement of 6.2 points. Although LLaVE is trained on image-text data, it can
generalize to text-video retrieval tasks in a zero-shot manner and achieve
strong performance, demonstrating its remarkable potential for transfer to
other embedding tasks.
The proposed LLaVE framework dynamically enhances embedding models' representation of negative pairs, achieving state-of-the-art performance across various multimodal tasks with strong scalability and efficiency.
LMM-based embedding models InfoNCE loss embedding model negative pairs discriminative difficulty LLaVE MMEB benchmark zero-shot manner