title: SEA: Supervised Embedding Alignment for Token-Level Visual-Textual
    Integration in MLLMs
summary: Multimodal Large Language Models (MLLMs) have recently demonstrated
  remarkable perceptual and reasoning abilities, typically comprising a Vision
  Encoder, an Adapter, and a Large Language Model (LLM). The adapter serves as
  the critical bridge between the visual and language components. However,
  training adapters with image-level supervision often results in significant
  misalignment, undermining the LLMs' capabilities and limiting the potential of
  Multimodal LLMs. To address this, we introduce Supervised Embedding Alignment
  (SEA), a token-level alignment method that leverages vision-language
  pre-trained models, such as CLIP, to align visual tokens with the LLM's
  embedding space through contrastive learning. This approach ensures a more
  coherent integration of visual and language representations, enhancing the
  performance and interpretability of multimodal LLMs while preserving their
  inherent capabilities. Extensive experiments show that SEA effectively improves
  MLLMs, particularly for smaller models, without adding extra data or inference
  computation. SEA also lays the groundwork for developing more general and
  adaptable solutions to enhance multimodal systems.
paper_authors._id: 66c6b7c04defcd6e2d34164e
paper_authors.name: Yuanyang Yin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-22T19:50:25.730Z
paper_authors._id: 659cb2671d398a2381625b2f
paper_authors.avatarUrl: /avatars/31ab5bf53fea8c9e6907aa3c921cc6ac.svg
paper_authors.fullname: YinYuanyang
paper_authors.type: user
paper_authors.user: yyyin
paper_authors._id: 66c6b7c04defcd6e2d34164f
paper_authors.name: Yaqi Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-22T19:50:23.342Z
paper_authors._id: 65b5cc2516301f4030d35db7
paper_authors.avatarUrl: /avatars/ba8d005c904ad3d8358fe69d518b51d7.svg
paper_authors.fullname: YaqiZhao
paper_authors.type: user
paper_authors.user: MiaTiancai
paper_authors._id: 66c6b7c04defcd6e2d341650
paper_authors.name: Yajie Zhang
paper_authors._id: 66c6b7c04defcd6e2d341651
paper_authors.name: Ke Lin
paper_authors._id: 66c6b7c04defcd6e2d341652
paper_authors.name: Jiahao Wang
paper_authors._id: 66c6b7c04defcd6e2d341653
paper_authors.name: Xin Tao
paper_authors._id: 66c6b7c04defcd6e2d341654
paper_authors.name: Pengfei Wan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-17T14:43:17.461Z
paper_authors._id: 662f93942510ef5735d7ad00
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
paper_authors.fullname: magicwpf
paper_authors.type: user
paper_authors.user: magicwpf
paper_authors._id: 66c6b7c04defcd6e2d341655
paper_authors.name: Di Zhang
paper_authors._id: 66c6b7c04defcd6e2d341656
paper_authors.name: Baoqun Yin
paper_authors._id: 66c6b7c04defcd6e2d341657
paper_authors.name: Wentao Zhang
paper_title: SEA: Supervised Embedding Alignment for Token-Level Visual-Textual
    Integration in MLLMs
paper_summary: Multimodal Large Language Models (MLLMs) have recently demonstrated
  remarkable perceptual and reasoning abilities, typically comprising a Vision
  Encoder, an Adapter, and a Large Language Model (LLM). The adapter serves as
  the critical bridge between the visual and language components. However,
  training adapters with image-level supervision often results in significant
  misalignment, undermining the LLMs' capabilities and limiting the potential of
  Multimodal LLMs. To address this, we introduce Supervised Embedding Alignment
  (SEA), a token-level alignment method that leverages vision-language
  pre-trained models, such as CLIP, to align visual tokens with the LLM's
  embedding space through contrastive learning. This approach ensures a more
  coherent integration of visual and language representations, enhancing the
  performance and interpretability of multimodal LLMs while preserving their
  inherent capabilities. Extensive experiments show that SEA effectively improves
  MLLMs, particularly for smaller models, without adding extra data or inference
  computation. SEA also lays the groundwork for developing more general and
  adaptable solutions to enhance multimodal systems.
paper_ai_summary: Supervised Embedding Alignment (SEA) improves Multimodal Large Language Models by aligning visual tokens with the LLM's embedding space, enhancing performance and interpretability.
paper_ai_keywords: Multimodal Large Language Models MLLMs Vision Encoder Adapter Large Language Model LLM token-level alignment vision-language pre-trained models CLIP contrastive learning visual tokens embedding space interpreted integration multimodal systems