SEA: Supervised Embedding Alignment for Token-Level Visual-Textual
  Integration in MLLMs
Multimodal Large Language Models (MLLMs) have recently demonstrated
remarkable perceptual and reasoning abilities, typically comprising a Vision
Encoder, an Adapter, and a Large Language Model (LLM). The adapter serves as
the critical bridge between the visual and language components. However,
training adapters with image-level supervision often results in significant
misalignment, undermining the LLMs' capabilities and limiting the potential of
Multimodal LLMs. To address this, we introduce Supervised Embedding Alignment
(SEA), a token-level alignment method that leverages vision-language
pre-trained models, such as CLIP, to align visual tokens with the LLM's
embedding space through contrastive learning. This approach ensures a more
coherent integration of visual and language representations, enhancing the
performance and interpretability of multimodal LLMs while preserving their
inherent capabilities. Extensive experiments show that SEA effectively improves
MLLMs, particularly for smaller models, without adding extra data or inference
computation. SEA also lays the groundwork for developing more general and
adaptable solutions to enhance multimodal systems.
66c6b7c04defcd6e2d34164e
Yuanyang Yin
claimed_verified
2024-08-22T19:50:25.730Z
659cb2671d398a2381625b2f
/avatars/31ab5bf53fea8c9e6907aa3c921cc6ac.svg
YinYuanyang
user
yyyin
66c6b7c04defcd6e2d34164f
Yaqi Zhao
claimed_verified
2024-08-22T19:50:23.342Z
65b5cc2516301f4030d35db7
/avatars/ba8d005c904ad3d8358fe69d518b51d7.svg
YaqiZhao
user
MiaTiancai
66c6b7c04defcd6e2d341650
Yajie Zhang
66c6b7c04defcd6e2d341651
Ke Lin
66c6b7c04defcd6e2d341652
Jiahao Wang
66c6b7c04defcd6e2d341653
Xin Tao
66c6b7c04defcd6e2d341654
Pengfei Wan
claimed_verified
2025-03-17T14:43:17.461Z
662f93942510ef5735d7ad00
https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
magicwpf
user
magicwpf
66c6b7c04defcd6e2d341655
Di Zhang
66c6b7c04defcd6e2d341656
Baoqun Yin
66c6b7c04defcd6e2d341657
Wentao Zhang
SEA: Supervised Embedding Alignment for Token-Level Visual-Textual
  Integration in MLLMs
Multimodal Large Language Models (MLLMs) have recently demonstrated
remarkable perceptual and reasoning abilities, typically comprising a Vision
Encoder, an Adapter, and a Large Language Model (LLM). The adapter serves as
the critical bridge between the visual and language components. However,
training adapters with image-level supervision often results in significant
misalignment, undermining the LLMs' capabilities and limiting the potential of
Multimodal LLMs. To address this, we introduce Supervised Embedding Alignment
(SEA), a token-level alignment method that leverages vision-language
pre-trained models, such as CLIP, to align visual tokens with the LLM's
embedding space through contrastive learning. This approach ensures a more
coherent integration of visual and language representations, enhancing the
performance and interpretability of multimodal LLMs while preserving their
inherent capabilities. Extensive experiments show that SEA effectively improves
MLLMs, particularly for smaller models, without adding extra data or inference
computation. SEA also lays the groundwork for developing more general and
adaptable solutions to enhance multimodal systems.
Supervised Embedding Alignment (SEA) improves Multimodal Large Language Models by aligning visual tokens with the LLM's embedding space, enhancing performance and interpretability.
Multimodal Large Language Models MLLMs Vision Encoder Adapter Large Language Model LLM token-level alignment vision-language pre-trained models CLIP contrastive learning visual tokens embedding space interpreted integration multimodal systems