title: ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language
    Tuning
summary: Recent advancements in multimodal fusion have witnessed the remarkable
  success of vision-language (VL) models, which excel in various multimodal
  applications such as image captioning and visual question answering. However,
  building VL models requires substantial hardware resources, where efficiency is
  restricted by two key factors: the extended input sequence of the language
  model with vision features demands more computational operations, and a large
  number of additional learnable parameters increase memory complexity. These
  challenges significantly restrict the broader applicability of such models. To
  bridge this gap, we propose ADEM-VL, an efficient vision-language method that
  tunes VL models based on pretrained large language models (LLMs) by adopting a
  parameter-free cross-attention mechanism for similarity measurements in
  multimodal fusion. This approach only requires embedding vision features into
  the language space, significantly reducing the number of trainable parameters
  and accelerating both training and inference speeds. To enhance representation
  learning in fusion module, we introduce an efficient multiscale feature
  generation scheme that requires only a single forward pass through the vision
  encoder. Moreover, we propose an adaptive fusion scheme that dynamically
  discards less relevant visual information for each text token based on its
  attention score. This ensures that the fusion process prioritizes the most
  pertinent visual features. With experiments on various tasks including visual
  question answering, image captioning, and instruction-following, we demonstrate
  that our framework outperforms existing approaches. Specifically, our method
  surpasses existing methods by an average accuracy of 0.77% on ScienceQA
  dataset, with reduced training and inference latency, demonstrating the
  superiority of our framework. The code is available at
  https://github.com/Hao840/ADEM-VL.
paper_authors._id: 671b45cc67d2d5784175afdc
paper_authors.name: Zhiwei Hao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:28:27.956Z
paper_authors._id: 64a62e3302e46deb19a7937e
paper_authors.avatarUrl: /avatars/43553a80f2c5f6c91742c4ce2d23fe21.svg
paper_authors.fullname: Zhiwei Hao
paper_authors.type: user
paper_authors.user: Zhiwei840
paper_authors._id: 671b45cc67d2d5784175afdd
paper_authors.name: Jianyuan Guo
paper_authors._id: 671b45cc67d2d5784175afde
paper_authors.name: Li Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-26T11:39:53.006Z
paper_authors._id: 62de356ad89af8c07209e7d4
paper_authors.avatarUrl: /avatars/610629958726b270418368b8b7f61469.svg
paper_authors.fullname: Li Shen
paper_authors.type: user
paper_authors.user: mathshenli
paper_authors._id: 671b45cc67d2d5784175afdf
paper_authors.name: Yong Luo
paper_authors._id: 671b45cc67d2d5784175afe0
paper_authors.name: Han Hu
paper_authors._id: 671b45cc67d2d5784175afe1
paper_authors.name: Yonggang Wen
paper_title: ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language
    Tuning
paper_summary: Recent advancements in multimodal fusion have witnessed the remarkable
  success of vision-language (VL) models, which excel in various multimodal
  applications such as image captioning and visual question answering. However,
  building VL models requires substantial hardware resources, where efficiency is
  restricted by two key factors: the extended input sequence of the language
  model with vision features demands more computational operations, and a large
  number of additional learnable parameters increase memory complexity. These
  challenges significantly restrict the broader applicability of such models. To
  bridge this gap, we propose ADEM-VL, an efficient vision-language method that
  tunes VL models based on pretrained large language models (LLMs) by adopting a
  parameter-free cross-attention mechanism for similarity measurements in
  multimodal fusion. This approach only requires embedding vision features into
  the language space, significantly reducing the number of trainable parameters
  and accelerating both training and inference speeds. To enhance representation
  learning in fusion module, we introduce an efficient multiscale feature
  generation scheme that requires only a single forward pass through the vision
  encoder. Moreover, we propose an adaptive fusion scheme that dynamically
  discards less relevant visual information for each text token based on its
  attention score. This ensures that the fusion process prioritizes the most
  pertinent visual features. With experiments on various tasks including visual
  question answering, image captioning, and instruction-following, we demonstrate
  that our framework outperforms existing approaches. Specifically, our method
  surpasses existing methods by an average accuracy of 0.77% on ScienceQA
  dataset, with reduced training and inference latency, demonstrating the
  superiority of our framework. The code is available at
  https://github.com/Hao840/ADEM-VL.
paper_ai_keywords: ADEM-VL vision-language (VL) models image captioning visual question answering pretrained large language models (LLMs) parameter-free cross-attention mechanism multimodal fusion embedding vision features into the language space multiscale feature generation scheme adaptive fusion scheme attention score instruction-following ScienceQA dataset training latency inference latency