title: QuickVideo: Real-Time Long Video Understanding with System Algorithm
    Co-Design
summary: Long-video understanding has emerged as a crucial capability in real-world
  applications such as video surveillance, meeting summarization, educational
  lecture analysis, and sports broadcasting. However, it remains computationally
  prohibitive for VideoLLMs, primarily due to two bottlenecks: 1) sequential
  video decoding, the process of converting the raw bit stream to RGB frames can
  take up to a minute for hour-long video inputs, and 2) costly prefilling of up
  to several million tokens for LLM inference, resulting in high latency and
  memory use. To address these challenges, we propose QuickVideo, a
  system-algorithm co-design that substantially accelerates long-video
  understanding to support real-time downstream applications. It comprises three
  key innovations: QuickDecoder, a parallelized CPU-based video decoder that
  achieves 2-3 times speedup by splitting videos into keyframe-aligned intervals
  processed concurrently; QuickPrefill, a memory-efficient prefilling method
  using KV-cache pruning to support more frames with less GPU memory; and an
  overlapping scheme that overlaps CPU video decoding with GPU inference.
  Together, these components infernece time reduce by a minute on long video
  inputs, enabling scalable, high-quality video understanding even on limited
  hardware. Experiments show that QuickVideo generalizes across durations and
  sampling rates, making long video processing feasible in practice.
paper_authors._id: 682fd91a1ffb93faf139d288
paper_authors.name: Benjamin Schneider
paper_authors._id: 682fd91a1ffb93faf139d289
paper_authors.name: Dongfu Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:16:44.817Z
paper_authors._id: 62567c86d444a9b5a0ec51c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62567c86d444a9b5a0ec51c1/1vXJf2uGztPcXpkwyTBr6.png
paper_authors.fullname: Dongfu Jiang
paper_authors.type: user
paper_authors.user: DongfuJiang
paper_authors._id: 682fd91a1ffb93faf139d28a
paper_authors.name: Chao Du
paper_authors._id: 682fd91a1ffb93faf139d28b
paper_authors.name: Tianyu Pang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:16:42.557Z
paper_authors._id: 63d91b6d255ef6add20e1b38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
paper_authors.fullname: Tianyu Pang
paper_authors.type: user
paper_authors.user: P2333
paper_authors._id: 682fd91a1ffb93faf139d28c
paper_authors.name: Wenhu Chen
paper_title: QuickVideo: Real-Time Long Video Understanding with System Algorithm
    Co-Design
paper_summary: Long-video understanding has emerged as a crucial capability in real-world
  applications such as video surveillance, meeting summarization, educational
  lecture analysis, and sports broadcasting. However, it remains computationally
  prohibitive for VideoLLMs, primarily due to two bottlenecks: 1) sequential
  video decoding, the process of converting the raw bit stream to RGB frames can
  take up to a minute for hour-long video inputs, and 2) costly prefilling of up
  to several million tokens for LLM inference, resulting in high latency and
  memory use. To address these challenges, we propose QuickVideo, a
  system-algorithm co-design that substantially accelerates long-video
  understanding to support real-time downstream applications. It comprises three
  key innovations: QuickDecoder, a parallelized CPU-based video decoder that
  achieves 2-3 times speedup by splitting videos into keyframe-aligned intervals
  processed concurrently; QuickPrefill, a memory-efficient prefilling method
  using KV-cache pruning to support more frames with less GPU memory; and an
  overlapping scheme that overlaps CPU video decoding with GPU inference.
  Together, these components infernece time reduce by a minute on long video
  inputs, enabling scalable, high-quality video understanding even on limited
  hardware. Experiments show that QuickVideo generalizes across durations and
  sampling rates, making long video processing feasible in practice.
paper_ai_summary: QuickVideo accelerates long-video understanding by combining a parallelized video decoder, memory-efficient prefilling, and overlapping video decoding with inference, enabling real-time performance.
paper_ai_keywords: QuickDecoder parallelized CPU-based video decoder keyframe-aligned intervals QuickPrefill memory-efficient prefilling KV-cache pruning overlapping scheme