SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models
<|reference_start|>SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models: We propose SlowFast-LLaVA (or SF-LLaVA for short), a training-free video large language model (LLM) that can jointly capture detailed spatial semantics and long-range temporal context without exceeding the token budget of commonly used LLMs. This is realized by using a two-stream SlowFast design of inputs for Video LLMs to aggregate features from sampled frames in an effective way. Specifically, the Slow pathway extracts features at a low frame rate while keeping as much spatial detail as possible (e.g., with 12x24 tokens), and the Fast pathway operates on a high frame rate but uses a larger spatial pooling stride (e.g., downsampling 6x) to focus on the motion cues. As a result, this design allows us to adequately capture both spatial and temporal features that are beneficial for detailed video understanding. Experimental results show that SF-LLaVA outperforms existing training-free methods on a wide range of video tasks. On some benchmarks, it achieves comparable or even better performance compared to state-of-the-art Video LLMs that are fine-tuned on video datasets. Code has been made available at: https://github.com/apple/ml-slowfast-llava.<|reference_end|>
@article{xu2024slowfast-llava:,
    title={SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language
  Models},
    author={Mingze Xu, Mingfei Gao, Zhe Gan, Hong-You Chen, Zhengfeng Lai, Haiming
  Gang, Kai Kang, Afshin Dehghan},
    journal={arXiv preprint arXiv:2407.15841},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.15841},
    primaryClass={cs.CV}
}