title: LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted
    Contrastive Learning
summary: Universal multimodal embedding models play a critical role in tasks such as
  interleaved image-text retrieval, multimodal RAG, and multimodal clustering.
  However, our empirical results indicate that existing LMM-based embedding
  models trained with the standard InfoNCE loss exhibit a high degree of overlap
  in similarity distribution between positive and negative pairs, making it
  challenging to distinguish hard negative pairs effectively. To deal with this
  issue, we propose a simple yet effective framework that dynamically improves
  the embedding model's representation learning for negative pairs based on their
  discriminative difficulty. Within this framework, we train a series of models,
  named LLaVE, and evaluate them on the MMEB benchmark, which covers 4 meta-tasks
  and 36 datasets. Experimental results show that LLaVE establishes stronger
  baselines that achieve state-of-the-art (SOTA) performance while demonstrating
  strong scalability and efficiency. Specifically, LLaVE-2B surpasses the
  previous SOTA 7B models, while LLaVE-7B achieves a further performance
  improvement of 6.2 points. Although LLaVE is trained on image-text data, it can
  generalize to text-video retrieval tasks in a zero-shot manner and achieve
  strong performance, demonstrating its remarkable potential for transfer to
  other embedding tasks.
paper_authors._id: 67ce5542818e1825dea7440b
paper_authors.name: Zhibin Lan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T08:00:54.535Z
paper_authors._id: 6626449503e1f561573d30e9
paper_authors.avatarUrl: /avatars/e7f9720ccd01bae32d0a03a1b0dacab5.svg
paper_authors.fullname: Zhibin Lan
paper_authors.type: user
paper_authors.user: zhibinlan
paper_authors._id: 67ce5542818e1825dea7440c
paper_authors.name: Liqiang Niu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T08:00:51.713Z
paper_authors._id: 635239137d071f23d083b056
paper_authors.avatarUrl: /avatars/1f1a0ed38d8de499d4b78922801c6d95.svg
paper_authors.fullname: liqiang niu
paper_authors.type: user
paper_authors.user: lqniu
paper_authors._id: 67ce5542818e1825dea7440d
paper_authors.name: Fandong Meng
paper_authors._id: 67ce5542818e1825dea7440e
paper_authors.name: Jie Zhou
paper_authors._id: 67ce5542818e1825dea7440f
paper_authors.name: Jinsong Su
paper_title: LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted
    Contrastive Learning
paper_summary: Universal multimodal embedding models play a critical role in tasks such as
  interleaved image-text retrieval, multimodal RAG, and multimodal clustering.
  However, our empirical results indicate that existing LMM-based embedding
  models trained with the standard InfoNCE loss exhibit a high degree of overlap
  in similarity distribution between positive and negative pairs, making it
  challenging to distinguish hard negative pairs effectively. To deal with this
  issue, we propose a simple yet effective framework that dynamically improves
  the embedding model's representation learning for negative pairs based on their
  discriminative difficulty. Within this framework, we train a series of models,
  named LLaVE, and evaluate them on the MMEB benchmark, which covers 4 meta-tasks
  and 36 datasets. Experimental results show that LLaVE establishes stronger
  baselines that achieve state-of-the-art (SOTA) performance while demonstrating
  strong scalability and efficiency. Specifically, LLaVE-2B surpasses the
  previous SOTA 7B models, while LLaVE-7B achieves a further performance
  improvement of 6.2 points. Although LLaVE is trained on image-text data, it can
  generalize to text-video retrieval tasks in a zero-shot manner and achieve
  strong performance, demonstrating its remarkable potential for transfer to
  other embedding tasks.
paper_ai_summary: The proposed LLaVE framework dynamically enhances embedding models' representation of negative pairs, achieving state-of-the-art performance across various multimodal tasks with strong scalability and efficiency.
paper_ai_keywords: LMM-based embedding models InfoNCE loss embedding model negative pairs discriminative difficulty LLaVE MMEB benchmark zero-shot manner