title: Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video
    Understanding
summary: We present Video-LLaMA, a multi-modal framework that empowers Large Language
  Models (LLMs) with the capability of understanding both visual and auditory
  content in the video. Video-LLaMA bootstraps cross-modal training from the
  frozen pre-trained visual \& audio encoders and the frozen LLMs. Unlike
  previous vision- LLMs that focus on static image comprehensions such as
  MiniGPT-4~zhu2023minigpt and LLaVA~liu2023visualit, Video-LLaMA
  tackles two challenges in video understanding: (1) capturing the temporal
  changes in visual scenes, (2) integrating audio-visual signals. For the first
  challenge, we propose Video Q-former to extend the pre-trained image encoder to
  a video encoder and introduce a video-to-text generation task to learn
  video-language correspondence. For the second challenge, we leverage
  ImageBind~girdhar2023imagebind as the pre-trained audio encoder which
  performs exceptionally well in aligning different modalities to a common
  embedding space. And then introduce an Audio Q-former to learn auditory query
  tokens. To align the output of both visual \& audio encoder with LLM's
  embedding space, we train Video-LLaMA on a large-scale vision caption dataset
  and a hign-quantity vision-instruction-tuning dataset. We found Video-LLaMA
  showcases the ability to perceive and comprehend video content, generating
  meaningful responses that are grounded in the visual and auditory information
  present in the videos. This highlights the potential of Video-LLaMA as a
  promising prototype for audio-visual AI assistants. Our code, pre-trained
  model, and demo are available at
  https://github.com/DAMO-NLP-SG/Video-LLaMA.
paper_authors._id: 647e9efc0ed7d0c8760edaf7
paper_authors.name: Hang Zhang
paper_authors._id: 647e9efc0ed7d0c8760edaf8
paper_authors.name: Xin Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-06T12:32:33.808Z
paper_authors._id: 63913b120cf6b11c487ca31d
paper_authors.avatarUrl: /avatars/aec44edd5470dd6e767e0a25efd6fb5d.svg
paper_authors.fullname: Xin Li
paper_authors.type: user
paper_authors.user: lixin4ever
paper_authors._id: 647e9efc0ed7d0c8760edaf9
paper_authors.name: Lidong Bing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-06T12:32:37.008Z
paper_authors._id: 6454685a548f22be598414c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
paper_authors.fullname: Lidong Bing
paper_authors.type: user
paper_authors.user: LidongBing
paper_title: Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video
    Understanding
paper_summary: We present Video-LLaMA, a multi-modal framework that empowers Large Language
  Models (LLMs) with the capability of understanding both visual and auditory
  content in the video. Video-LLaMA bootstraps cross-modal training from the
  frozen pre-trained visual \& audio encoders and the frozen LLMs. Unlike
  previous vision- LLMs that focus on static image comprehensions such as
  MiniGPT-4~zhu2023minigpt and LLaVA~liu2023visualit, Video-LLaMA
  tackles two challenges in video understanding: (1) capturing the temporal
  changes in visual scenes, (2) integrating audio-visual signals. For the first
  challenge, we propose Video Q-former to extend the pre-trained image encoder to
  a video encoder and introduce a video-to-text generation task to learn
  video-language correspondence. For the second challenge, we leverage
  ImageBind~girdhar2023imagebind as the pre-trained audio encoder which
  performs exceptionally well in aligning different modalities to a common
  embedding space. And then introduce an Audio Q-former to learn auditory query
  tokens. To align the output of both visual \& audio encoder with LLM's
  embedding space, we train Video-LLaMA on a large-scale vision caption dataset
  and a hign-quantity vision-instruction-tuning dataset. We found Video-LLaMA
  showcases the ability to perceive and comprehend video content, generating
  meaningful responses that are grounded in the visual and auditory information
  present in the videos. This highlights the potential of Video-LLaMA as a
  promising prototype for audio-visual AI assistants. Our code, pre-trained
  model, and demo are available at
  https://github.com/DAMO-NLP-SG/Video-LLaMA.
paper_ai_keywords: Video Q-former video encoder video-to-text generation ImageBind Audio Q-former auditory query tokens vision caption dataset vision-instruction-tuning dataset large-scale vision caption dataset