title: VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio
    Understanding in Video-LLMs
summary: In this paper, we present the VideoLLaMA 2, a set of Video Large Language
  Models (Video-LLMs) designed to enhance spatial-temporal modeling and audio
  understanding in video and audio-oriented tasks. Building upon its predecessor,
  VideoLLaMA 2 incorporates a tailor-made Spatial-Temporal Convolution (STC)
  connector, which effectively captures the intricate spatial and temporal
  dynamics of video data. Additionally, we integrate an Audio Branch into the
  model through joint training, thereby enriching the multimodal understanding
  capabilities of the model by seamlessly incorporating audio cues. Comprehensive
  evaluations on multiple-choice video question answering (MC-VQA), open-ended
  video question answering (OE-VQA), and video captioning (VC) tasks demonstrate
  that VideoLLaMA 2 consistently achieves competitive results among open-source
  models and even gets close to some proprietary models on several benchmarks.
  Furthermore, VideoLLaMA 2 exhibits reasonable improvements in audio-only and
  audio-video question-answering (AQA & OE-AVQA) benchmarks over existing models.
  These advancements underline VideoLLaMA 2's superior performance in multimodal
  comprehension, setting a new standard for intelligent video analysis systems.
  All models are public to facilitate further research.
paper_authors._id: 66692323d993003c25001828
paper_authors.name: Zesen Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T13:05:24.153Z
paper_authors._id: 65b2529285b6c21448a10d65
paper_authors.avatarUrl: /avatars/1b09e2742aecce1bbdc57f0c4504cf38.svg
paper_authors.fullname: Zesen Cheng
paper_authors.type: user
paper_authors.user: ClownRat
paper_authors._id: 66692323d993003c25001829
paper_authors.name: Sicong Leng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T13:05:34.956Z
paper_authors._id: 609115c79a8bcaa437b234a9
paper_authors.avatarUrl: /avatars/1631a91030703d8397133363cf82c863.svg
paper_authors.fullname: Leng Sicong
paper_authors.type: user
paper_authors.user: Sicong
paper_authors._id: 66692323d993003c2500182a
paper_authors.name: Hang Zhang
paper_authors._id: 66692323d993003c2500182b
paper_authors.name: Yifei Xin
paper_authors._id: 66692323d993003c2500182c
paper_authors.name: Xin Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:31:47.064Z
paper_authors._id: 63913b120cf6b11c487ca31d
paper_authors.avatarUrl: /avatars/aec44edd5470dd6e767e0a25efd6fb5d.svg
paper_authors.fullname: Xin Li
paper_authors.type: user
paper_authors.user: lixin4ever
paper_authors._id: 66692323d993003c2500182d
paper_authors.name: Guanzheng Chen
paper_authors._id: 66692323d993003c2500182e
paper_authors.name: Yongxin Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T07:41:53.168Z
paper_authors._id: 637c7d8a88699fba70e1e1ff
paper_authors.avatarUrl: /avatars/e7f60fab6ca9ad5ecd320bd7cd51ce2e.svg
paper_authors.fullname: yongxinzhu
paper_authors.type: user
paper_authors.user: youngsheen
paper_authors._id: 66692323d993003c2500182f
paper_authors.name: Wenqi Zhang
paper_authors._id: 66692323d993003c25001830
paper_authors.name: Ziyang Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T13:07:20.220Z
paper_authors._id: 6090ff099a8bcaa437b234a4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6090ff099a8bcaa437b234a4/iUvw7JXT-ngI7rGk1x-io.jpeg
paper_authors.fullname: Ziyang Luo
paper_authors.type: user
paper_authors.user: Ziyang
paper_authors._id: 66692323d993003c25001831
paper_authors.name: Deli Zhao
paper_authors._id: 66692323d993003c25001832
paper_authors.name: Lidong Bing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T13:07:33.945Z
paper_authors._id: 6454685a548f22be598414c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
paper_authors.fullname: Lidong Bing
paper_authors.type: user
paper_authors.user: LidongBing
paper_title: VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio
    Understanding in Video-LLMs
paper_summary: In this paper, we present the VideoLLaMA 2, a set of Video Large Language
  Models (Video-LLMs) designed to enhance spatial-temporal modeling and audio
  understanding in video and audio-oriented tasks. Building upon its predecessor,
  VideoLLaMA 2 incorporates a tailor-made Spatial-Temporal Convolution (STC)
  connector, which effectively captures the intricate spatial and temporal
  dynamics of video data. Additionally, we integrate an Audio Branch into the
  model through joint training, thereby enriching the multimodal understanding
  capabilities of the model by seamlessly incorporating audio cues. Comprehensive
  evaluations on multiple-choice video question answering (MC-VQA), open-ended
  video question answering (OE-VQA), and video captioning (VC) tasks demonstrate
  that VideoLLaMA 2 consistently achieves competitive results among open-source
  models and even gets close to some proprietary models on several benchmarks.
  Furthermore, VideoLLaMA 2 exhibits reasonable improvements in audio-only and
  audio-video question-answering (AQA & OE-AVQA) benchmarks over existing models.
  These advancements underline VideoLLaMA 2's superior performance in multimodal
  comprehension, setting a new standard for intelligent video analysis systems.
  All models are public to facilitate further research.
paper_ai_summary: VideoLLaMA 2, an enhanced Video Large Language Model, incorporates Spatial-Temporal Convolution and audio cues to achieve competitive performance in multimodal video and audio tasks.
paper_ai_keywords: Spatial-Temporal Convolution Video Large Language Models Video-LLMs multimodal understanding multiple-choice video question answering open-ended video question answering video captioning audio-only question answering audio-video question-answering