title: PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong
    Vision-language Adapter
summary: This paper demonstrates that a progressively aligned language model can
  effectively bridge frozen vision encoders and large language models (LLMs).
  While the fundamental architecture and pre-training methods of vision encoders
  and LLMs have been extensively studied, the architecture and training strategy
  of vision-language adapters vary significantly across recent works. Our
  research undertakes a thorough exploration of the state-of-the-art perceiver
  resampler architecture and builds a strong baseline. However, we observe that
  the vision-language alignment with perceiver resampler exhibits slow
  convergence and limited scalability with a lack of direct supervision. To
  address this issue, we propose PaLM2-VAdapter, employing a progressively
  aligned language model as the vision-language adapter. Compared to the strong
  baseline with perceiver resampler, our method empirically shows faster
  convergence, higher performance, and stronger scalability. Extensive
  experiments across various Visual Question Answering (VQA) and captioning tasks
  on both images and videos demonstrate that our model exhibits state-of-the-art
  visual understanding and multi-modal reasoning capabilities. Notably, our
  method achieves these advancements with 30~70% fewer parameters than the
  state-of-the-art large vision-language models, marking a significant efficiency
  improvement.
paper_authors._id: 65d2e7b5ff101ee25e170c66
paper_authors.name: Junfei Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-19T15:03:15.371Z
paper_authors._id: 64b5ba6060274cbb296d6288
paper_authors.avatarUrl: /avatars/67e0343954dda6e92ed3f6e7976f9f87.svg
paper_authors.fullname: Junfei Xiao
paper_authors.type: user
paper_authors.user: lambertxiao
paper_authors._id: 65d2e7b5ff101ee25e170c67
paper_authors.name: Zheng Xu
paper_authors._id: 65d2e7b5ff101ee25e170c68
paper_authors.name: Alan Yuille
paper_authors._id: 65d2e7b5ff101ee25e170c69
paper_authors.name: Shen Yan
paper_authors._id: 65d2e7b5ff101ee25e170c6a
paper_authors.name: Boyu Wang
paper_title: PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong
    Vision-language Adapter
paper_summary: This paper demonstrates that a progressively aligned language model can
  effectively bridge frozen vision encoders and large language models (LLMs).
  While the fundamental architecture and pre-training methods of vision encoders
  and LLMs have been extensively studied, the architecture and training strategy
  of vision-language adapters vary significantly across recent works. Our
  research undertakes a thorough exploration of the state-of-the-art perceiver
  resampler architecture and builds a strong baseline. However, we observe that
  the vision-language alignment with perceiver resampler exhibits slow
  convergence and limited scalability with a lack of direct supervision. To
  address this issue, we propose PaLM2-VAdapter, employing a progressively
  aligned language model as the vision-language adapter. Compared to the strong
  baseline with perceiver resampler, our method empirically shows faster
  convergence, higher performance, and stronger scalability. Extensive
  experiments across various Visual Question Answering (VQA) and captioning tasks
  on both images and videos demonstrate that our model exhibits state-of-the-art
  visual understanding and multi-modal reasoning capabilities. Notably, our
  method achieves these advancements with 30~70% fewer parameters than the
  state-of-the-art large vision-language models, marking a significant efficiency
  improvement.
paper_ai_keywords: progressively aligned language model vision encoders large language models (LLMs) perceiver resampler architecture vision-language adapters convergence scalability direct supervision PaLM2-VAdapter Visual Question Answering (VQA) captioning visual understanding multi-modal reasoning parameter-efficient