title: Video-Panda: Parameter-efficient Alignment for Encoder-free
    Video-Language Models
summary: We present an efficient encoder-free approach for video-language
  understanding that achieves competitive performance while significantly
  reducing computational overhead. Current video-language models typically rely
  on heavyweight image encoders (300M-1.1B parameters) or video encoders (1B-1.4B
  parameters), creating a substantial computational burden when processing
  multi-frame videos. Our method introduces a novel Spatio-Temporal Alignment
  Block (STAB) that directly processes video inputs without requiring pre-trained
  encoders while using only 45M parameters for visual processing - at least a
  6.5times reduction compared to traditional approaches. The STAB architecture
  combines Local Spatio-Temporal Encoding for fine-grained feature extraction,
  efficient spatial downsampling through learned attention and separate
  mechanisms for modeling frame-level and video-level relationships. Our model
  achieves comparable or superior performance to encoder-based approaches for
  open-ended video question answering on standard benchmarks. The fine-grained
  video question-answering evaluation demonstrates our model's effectiveness,
  outperforming the encoder-based approaches Video-ChatGPT and Video-LLaVA in key
  aspects like correctness and temporal understanding. Extensive ablation studies
  validate our architectural choices and demonstrate the effectiveness of our
  spatio-temporal modeling approach while achieving 3-4times faster processing
  speeds than previous methods. Code is available at
  https://github.com/jh-yi/Video-Panda.
paper_authors._id: 676d854b8771d55751ee0f4f
paper_authors.name: Jinhui Yi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:31:40.301Z
paper_authors._id: 667dfe6f684550b7893497c5
paper_authors.avatarUrl: /avatars/3f48b041a826fd8a8cc74c2d4ced1705.svg
paper_authors.fullname: Jinhui Yi
paper_authors.type: user
paper_authors.user: jh-yi
paper_authors._id: 676d854b8771d55751ee0f50
paper_authors.name: Syed Talal Wasim
paper_authors._id: 676d854b8771d55751ee0f51
paper_authors.name: Yanan Luo
paper_authors._id: 676d854b8771d55751ee0f52
paper_authors.name: Muzammal Naseer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T21:17:02.313Z
paper_authors._id: 6495aae2e9b6d89a5c65cf15
paper_authors.avatarUrl: /avatars/c176d68af77698745c23da8b92424738.svg
paper_authors.fullname: Muzammal Naseer
paper_authors.type: user
paper_authors.user: muzammal
paper_authors._id: 676d854b8771d55751ee0f53
paper_authors.name: Juergen Gall
paper_title: Video-Panda: Parameter-efficient Alignment for Encoder-free
    Video-Language Models
paper_summary: We present an efficient encoder-free approach for video-language
  understanding that achieves competitive performance while significantly
  reducing computational overhead. Current video-language models typically rely
  on heavyweight image encoders (300M-1.1B parameters) or video encoders (1B-1.4B
  parameters), creating a substantial computational burden when processing
  multi-frame videos. Our method introduces a novel Spatio-Temporal Alignment
  Block (STAB) that directly processes video inputs without requiring pre-trained
  encoders while using only 45M parameters for visual processing - at least a
  6.5times reduction compared to traditional approaches. The STAB architecture
  combines Local Spatio-Temporal Encoding for fine-grained feature extraction,
  efficient spatial downsampling through learned attention and separate
  mechanisms for modeling frame-level and video-level relationships. Our model
  achieves comparable or superior performance to encoder-based approaches for
  open-ended video question answering on standard benchmarks. The fine-grained
  video question-answering evaluation demonstrates our model's effectiveness,
  outperforming the encoder-based approaches Video-ChatGPT and Video-LLaVA in key
  aspects like correctness and temporal understanding. Extensive ablation studies
  validate our architectural choices and demonstrate the effectiveness of our
  spatio-temporal modeling approach while achieving 3-4times faster processing
  speeds than previous methods. Code is available at
  https://github.com/jh-yi/Video-Panda.
paper_ai_summary: An encoder-free approach for video-language understanding using Spatio-Temporal Alignment Blocks achieves competitive performance with significantly reduced computational overhead and faster processing speeds.
paper_ai_keywords: Spatio-Temporal Alignment Block STAB Local Spatio-Temporal Encoding learned attention frame-level relationships video-level relationships video question answering Video-ChatGPT Video-LLaVA