title: Slow-Fast Architecture for Video Multi-Modal Large Language Models
summary: Balancing temporal resolution and spatial detail under limited compute budget
  remains a key challenge for video-based multi-modal large language models
  (MLLMs). Existing methods typically compress video representations using
  predefined rules before feeding them into the LLM, resulting in irreversible
  information loss and often ignoring input instructions. To address this, we
  propose a novel slow-fast architecture that naturally circumvents this
  trade-off, enabling the use of more input frames while preserving spatial
  details. Inspired by how humans first skim a video before focusing on relevant
  parts, our slow-fast design employs a dual-token strategy: 1) "fast" visual
  tokens -- a compact set of compressed video features -- are fed into the LLM
  alongside text embeddings to provide a quick overview; 2) "slow" visual tokens
  -- uncompressed video features -- are cross-attended by text embeddings through
  specially designed hybrid decoder layers, enabling instruction-aware extraction
  of relevant visual details with linear complexity. We conduct systematic
  exploration to optimize both the overall architecture and key components.
  Experiments show that our model significantly outperforms self-attention-only
  baselines, extending the input capacity from 16 to 128 frames with just a 3%
  increase in computation, and achieving a 16% average performance improvement
  across five video understanding benchmarks. Our 7B model achieves
  state-of-the-art performance among models of similar size. Furthermore, our
  slow-fast architecture is a plug-and-play design that can be integrated into
  other video MLLMs to improve efficiency and scalability.
paper_authors._id: 67f3cc0bbb7b1ce2a0a428f1
paper_authors.name: Min Shi
paper_authors._id: 67f3cc0bbb7b1ce2a0a428f2
paper_authors.name: Shihao Wang
paper_authors._id: 67f3cc0bbb7b1ce2a0a428f3
paper_authors.name: Chieh-Yun Chen
paper_authors._id: 67f3cc0bbb7b1ce2a0a428f4
paper_authors.name: Jitesh Jain
paper_authors._id: 67f3cc0bbb7b1ce2a0a428f5
paper_authors.name: Kai Wang
paper_authors._id: 67f3cc0bbb7b1ce2a0a428f6
paper_authors.name: Junjun Xiong
paper_authors._id: 67f3cc0bbb7b1ce2a0a428f7
paper_authors.name: Guilin Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T10:15:39.285Z
paper_authors._id: 63bc4115141c7d395c4aa02c
paper_authors.avatarUrl: /avatars/a9eca59e8c52cceab6524a4aaab11e38.svg
paper_authors.fullname: Liu
paper_authors.type: user
paper_authors.user: liuguilin
paper_authors._id: 67f3cc0bbb7b1ce2a0a428f8
paper_authors.name: Zhiding Yu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-04-07T12:58:56.336Z
paper_authors._id: 66c8037c737ba92ae3fe0322
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66c8037c737ba92ae3fe0322/WR_Yh5DWOVVh7IFlF24NM.jpeg
paper_authors.fullname: Zhiding Yu
paper_authors.type: user
paper_authors.user: Zhiding
paper_authors._id: 67f3cc0bbb7b1ce2a0a428f9
paper_authors.name: Humphrey Shi
paper_title: Slow-Fast Architecture for Video Multi-Modal Large Language Models
paper_summary: Balancing temporal resolution and spatial detail under limited compute budget
  remains a key challenge for video-based multi-modal large language models
  (MLLMs). Existing methods typically compress video representations using
  predefined rules before feeding them into the LLM, resulting in irreversible
  information loss and often ignoring input instructions. To address this, we
  propose a novel slow-fast architecture that naturally circumvents this
  trade-off, enabling the use of more input frames while preserving spatial
  details. Inspired by how humans first skim a video before focusing on relevant
  parts, our slow-fast design employs a dual-token strategy: 1) "fast" visual
  tokens -- a compact set of compressed video features -- are fed into the LLM
  alongside text embeddings to provide a quick overview; 2) "slow" visual tokens
  -- uncompressed video features -- are cross-attended by text embeddings through
  specially designed hybrid decoder layers, enabling instruction-aware extraction
  of relevant visual details with linear complexity. We conduct systematic
  exploration to optimize both the overall architecture and key components.
  Experiments show that our model significantly outperforms self-attention-only
  baselines, extending the input capacity from 16 to 128 frames with just a 3%
  increase in computation, and achieving a 16% average performance improvement
  across five video understanding benchmarks. Our 7B model achieves
  state-of-the-art performance among models of similar size. Furthermore, our
  slow-fast architecture is a plug-and-play design that can be integrated into
  other video MLLMs to improve efficiency and scalability.
paper_ai_summary: A slow-fast architecture using dual-token strategy enables efficient and high-performing video-based multi-modal large language models by balancing temporal resolution and spatial detail.
paper_ai_keywords: slow-fast architecture dual-token strategy fast visual tokens slow visual tokens cross-attention hybrid decoder layers video-based multi-modal large language models self-attention video understanding benchmarks