title: E5-V: Universal Embeddings with Multimodal Large Language Models
summary: Multimodal large language models (MLLMs) have shown promising advancements in
  general visual and language understanding. However, the representation of
  multimodal information using MLLMs remains largely unexplored. In this work, we
  introduce a new framework, E5-V, designed to adapt MLLMs for achieving
  universal multimodal embeddings. Our findings highlight the significant
  potential of MLLMs in representing multimodal inputs compared to previous
  approaches. By leveraging MLLMs with prompts, E5-V effectively bridges the
  modality gap between different types of inputs, demonstrating strong
  performance in multimodal embeddings even without fine-tuning. We propose a
  single modality training approach for E5-V, where the model is trained
  exclusively on text pairs. This method demonstrates significant improvements
  over traditional multimodal training on image-text pairs, while reducing
  training costs by approximately 95%. Additionally, this approach eliminates the
  need for costly multimodal training data collection. Extensive experiments
  across four types of tasks demonstrate the effectiveness of E5-V. As a
  universal multimodal model, E5-V not only achieves but often surpasses
  state-of-the-art performance in each task, despite being trained on a single
  modality.
paper_authors._id: 66987f3cbc2f8c7459128bf4
paper_authors.name: Ting Jiang
paper_authors._id: 66987f3cbc2f8c7459128bf5
paper_authors.name: Minghui Song
paper_authors._id: 66987f3cbc2f8c7459128bf6
paper_authors.name: Zihan Zhang
paper_authors._id: 66987f3cbc2f8c7459128bf7
paper_authors.name: Haizhen Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T07:10:45.003Z
paper_authors._id: 6302e8e57e137e3d6b3f5a06
paper_authors.avatarUrl: /avatars/aa82fe4fb1212dc120ecccd406568c5d.svg
paper_authors.fullname: Huang
paper_authors.type: user
paper_authors.user: Haizhen
paper_authors._id: 66987f3cbc2f8c7459128bf8
paper_authors.name: Weiwei Deng
paper_authors._id: 66987f3cbc2f8c7459128bf9
paper_authors.name: Feng Sun
paper_authors._id: 66987f3cbc2f8c7459128bfa
paper_authors.name: Qi Zhang
paper_authors._id: 66987f3cbc2f8c7459128bfb
paper_authors.name: Deqing Wang
paper_authors._id: 66987f3cbc2f8c7459128bfc
paper_authors.name: Fuzhen Zhuang
paper_title: E5-V: Universal Embeddings with Multimodal Large Language Models
paper_summary: Multimodal large language models (MLLMs) have shown promising advancements in
  general visual and language understanding. However, the representation of
  multimodal information using MLLMs remains largely unexplored. In this work, we
  introduce a new framework, E5-V, designed to adapt MLLMs for achieving
  universal multimodal embeddings. Our findings highlight the significant
  potential of MLLMs in representing multimodal inputs compared to previous
  approaches. By leveraging MLLMs with prompts, E5-V effectively bridges the
  modality gap between different types of inputs, demonstrating strong
  performance in multimodal embeddings even without fine-tuning. We propose a
  single modality training approach for E5-V, where the model is trained
  exclusively on text pairs. This method demonstrates significant improvements
  over traditional multimodal training on image-text pairs, while reducing
  training costs by approximately 95%. Additionally, this approach eliminates the
  need for costly multimodal training data collection. Extensive experiments
  across four types of tasks demonstrate the effectiveness of E5-V. As a
  universal multimodal model, E5-V not only achieves but often surpasses
  state-of-the-art performance in each task, despite being trained on a single
  modality.
paper_ai_keywords: neural large language models (MLLMs) visual and language understanding universal multimodal embeddings E5-V framework prompts modality gap single modality training approach text pairs traditional multimodal training multimodal training data collection universal multimodal model state-of-the-art performance