Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video
  Understanding
We present Video-LLaMA, a multi-modal framework that empowers Large Language
Models (LLMs) with the capability of understanding both visual and auditory
content in the video. Video-LLaMA bootstraps cross-modal training from the
frozen pre-trained visual \& audio encoders and the frozen LLMs. Unlike
previous vision- LLMs that focus on static image comprehensions such as
MiniGPT-4~zhu2023minigpt and LLaVA~liu2023visualit, Video-LLaMA
tackles two challenges in video understanding: (1) capturing the temporal
changes in visual scenes, (2) integrating audio-visual signals. For the first
challenge, we propose Video Q-former to extend the pre-trained image encoder to
a video encoder and introduce a video-to-text generation task to learn
video-language correspondence. For the second challenge, we leverage
ImageBind~girdhar2023imagebind as the pre-trained audio encoder which
performs exceptionally well in aligning different modalities to a common
embedding space. And then introduce an Audio Q-former to learn auditory query
tokens. To align the output of both visual \& audio encoder with LLM's
embedding space, we train Video-LLaMA on a large-scale vision caption dataset
and a hign-quantity vision-instruction-tuning dataset. We found Video-LLaMA
showcases the ability to perceive and comprehend video content, generating
meaningful responses that are grounded in the visual and auditory information
present in the videos. This highlights the potential of Video-LLaMA as a
promising prototype for audio-visual AI assistants. Our code, pre-trained
model, and demo are available at
https://github.com/DAMO-NLP-SG/Video-LLaMA.
647e9efc0ed7d0c8760edaf7
Hang Zhang
647e9efc0ed7d0c8760edaf8
Xin Li
claimed_verified
2023-06-06T12:32:33.808Z
63913b120cf6b11c487ca31d
/avatars/aec44edd5470dd6e767e0a25efd6fb5d.svg
Xin Li
user
lixin4ever
647e9efc0ed7d0c8760edaf9
Lidong Bing
claimed_verified
2023-06-06T12:32:37.008Z
6454685a548f22be598414c4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
Lidong Bing
user
LidongBing
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video
  Understanding
We present Video-LLaMA, a multi-modal framework that empowers Large Language
Models (LLMs) with the capability of understanding both visual and auditory
content in the video. Video-LLaMA bootstraps cross-modal training from the
frozen pre-trained visual \& audio encoders and the frozen LLMs. Unlike
previous vision- LLMs that focus on static image comprehensions such as
MiniGPT-4~zhu2023minigpt and LLaVA~liu2023visualit, Video-LLaMA
tackles two challenges in video understanding: (1) capturing the temporal
changes in visual scenes, (2) integrating audio-visual signals. For the first
challenge, we propose Video Q-former to extend the pre-trained image encoder to
a video encoder and introduce a video-to-text generation task to learn
video-language correspondence. For the second challenge, we leverage
ImageBind~girdhar2023imagebind as the pre-trained audio encoder which
performs exceptionally well in aligning different modalities to a common
embedding space. And then introduce an Audio Q-former to learn auditory query
tokens. To align the output of both visual \& audio encoder with LLM's
embedding space, we train Video-LLaMA on a large-scale vision caption dataset
and a hign-quantity vision-instruction-tuning dataset. We found Video-LLaMA
showcases the ability to perceive and comprehend video content, generating
meaningful responses that are grounded in the visual and auditory information
present in the videos. This highlights the potential of Video-LLaMA as a
promising prototype for audio-visual AI assistants. Our code, pre-trained
model, and demo are available at
https://github.com/DAMO-NLP-SG/Video-LLaMA.
Video-LLaMA is a multi-modal framework that integrates visual and auditory understanding into LLMs by extending image encoders to video and aligning audio signals, achieving effective video-language correspondence.
Video-LLaMA multi-modal framework Large Language Models (LLMs) cross-modal training visual encoder audio encoder Video Q-former Audio Q-former video-to-text generation ImageBind vision caption dataset vision-instruction-tuning dataset audio-visual AI assistants