title: SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language
    Models
summary: We propose SlowFast-LLaVA (or SF-LLaVA for short), a training-free video
  large language model (LLM) that can jointly capture the detailed spatial
  semantics and long-range temporal context without exceeding the token budget of
  commonly used LLMs. This is realized by using a two-stream SlowFast design of
  inputs for Video LLMs to aggregate features from sampled video frames in an
  effective way. Specifically, the Slow pathway extracts features at a low frame
  rate while keeping as many spatial details as possible (e.g., with 24x24
  tokens), and the Fast pathway operates on a high frame rate but uses a larger
  spatial pooling stride (e.g., downsampling 6x) to focus on the motion cues. As
  a result, this design allows us to adequately capture both spatial and temporal
  features that are beneficial for understanding details along the video.
  Experimental results show that SF-LLaVA outperforms existing training-free
  methods on a wide range of video tasks. On some benchmarks, it achieves
  comparable or even better performance compared to state-of-the-art Video LLMs
  that are fine-tuned on video datasets.
paper_authors._id: 669f21b88a6fac322926ded2
paper_authors.name: Mingze Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-24T07:27:25.835Z
paper_authors._id: 65fc655930aee594d74dd7b8
paper_authors.avatarUrl: /avatars/e8df7170fa1cf99bb3a2b9a083ee99a1.svg
paper_authors.fullname: Mingze Xu
paper_authors.type: user
paper_authors.user: xumingze0308
paper_authors._id: 669f21b88a6fac322926ded3
paper_authors.name: Mingfei Gao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-24T07:27:23.525Z
paper_authors._id: 64388d5f1efe72ba48071c52
paper_authors.avatarUrl: /avatars/2653fba13e71f471e9d55c134fe25efc.svg
paper_authors.fullname: Mingfei Gao
paper_authors.type: user
paper_authors.user: fly6464
paper_authors._id: 669f21b88a6fac322926ded4
paper_authors.name: Zhe Gan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T08:01:34.704Z
paper_authors._id: 644022db21ec7c165d848b6e
paper_authors.avatarUrl: /avatars/d9ed37ec77f2682878b2976a04d2fcd8.svg
paper_authors.fullname: Zhe Gan
paper_authors.type: user
paper_authors.user: zhegan27
paper_authors._id: 669f21b88a6fac322926ded5
paper_authors.name: Hong-You Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T08:01:41.355Z
paper_authors._id: 64b720891a4d97b5d7f055b8
paper_authors.avatarUrl: /avatars/7247a4f3a433a58fe934f245d9a757b2.svg
paper_authors.fullname: Hong-You Chen
paper_authors.type: user
paper_authors.user: Cbbjames
paper_authors._id: 669f21b88a6fac322926ded6
paper_authors.name: Zhengfeng Lai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-09T07:48:33.573Z
paper_authors._id: 66b5295f83425904fa7a1a6a
paper_authors.avatarUrl: /avatars/a35568fb933ceef7451bd88fb3d5ab17.svg
paper_authors.fullname: Zhengfeng Lai
paper_authors.type: user
paper_authors.user: jefflai
paper_authors._id: 669f21b88a6fac322926ded7
paper_authors.name: Haiming Gang
paper_authors._id: 669f21b88a6fac322926ded8
paper_authors.name: Kai Kang
paper_authors._id: 669f21b88a6fac322926ded9
paper_authors.name: Afshin Dehghan
paper_title: SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language
    Models
paper_summary: We propose SlowFast-LLaVA (or SF-LLaVA for short), a training-free video
  large language model (LLM) that can jointly capture the detailed spatial
  semantics and long-range temporal context without exceeding the token budget of
  commonly used LLMs. This is realized by using a two-stream SlowFast design of
  inputs for Video LLMs to aggregate features from sampled video frames in an
  effective way. Specifically, the Slow pathway extracts features at a low frame
  rate while keeping as many spatial details as possible (e.g., with 24x24
  tokens), and the Fast pathway operates on a high frame rate but uses a larger
  spatial pooling stride (e.g., downsampling 6x) to focus on the motion cues. As
  a result, this design allows us to adequately capture both spatial and temporal
  features that are beneficial for understanding details along the video.
  Experimental results show that SF-LLaVA outperforms existing training-free
  methods on a wide range of video tasks. On some benchmarks, it achieves
  comparable or even better performance compared to state-of-the-art Video LLMs
  that are fine-tuned on video datasets.
paper_ai_summary: SlowFast-LLaVA captures detailed spatial and temporal features in videos using a two-stream design, outperforming existing training-free methods and matching fine-tuned Video LLMs.
paper_ai_keywords: SlowFast video large language model two-stream design Slow pathway Fast pathway frame rate spatial details temporal context motion cues training-free video tasks state-of-the-art Video LLMs