QuickVideo: Real-Time Long Video Understanding with System Algorithm
  Co-Design
Long-video understanding has emerged as a crucial capability in real-world
applications such as video surveillance, meeting summarization, educational
lecture analysis, and sports broadcasting. However, it remains computationally
prohibitive for VideoLLMs, primarily due to two bottlenecks: 1) sequential
video decoding, the process of converting the raw bit stream to RGB frames can
take up to a minute for hour-long video inputs, and 2) costly prefilling of up
to several million tokens for LLM inference, resulting in high latency and
memory use. To address these challenges, we propose QuickVideo, a
system-algorithm co-design that substantially accelerates long-video
understanding to support real-time downstream applications. It comprises three
key innovations: QuickDecoder, a parallelized CPU-based video decoder that
achieves 2-3 times speedup by splitting videos into keyframe-aligned intervals
processed concurrently; QuickPrefill, a memory-efficient prefilling method
using KV-cache pruning to support more frames with less GPU memory; and an
overlapping scheme that overlaps CPU video decoding with GPU inference.
Together, these components infernece time reduce by a minute on long video
inputs, enabling scalable, high-quality video understanding even on limited
hardware. Experiments show that QuickVideo generalizes across durations and
sampling rates, making long video processing feasible in practice.
682fd91a1ffb93faf139d288
Benjamin Schneider
682fd91a1ffb93faf139d289
Dongfu Jiang
claimed_verified
2025-05-26T08:16:44.817Z
62567c86d444a9b5a0ec51c1
https://cdn-avatars.huggingface.co/v1/production/uploads/62567c86d444a9b5a0ec51c1/1vXJf2uGztPcXpkwyTBr6.png
Dongfu Jiang
user
DongfuJiang
682fd91a1ffb93faf139d28a
Chao Du
682fd91a1ffb93faf139d28b
Tianyu Pang
claimed_verified
2025-05-26T08:16:42.557Z
63d91b6d255ef6add20e1b38
https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
Tianyu Pang
user
P2333
682fd91a1ffb93faf139d28c
Wenhu Chen
QuickVideo: Real-Time Long Video Understanding with System Algorithm
  Co-Design
Long-video understanding has emerged as a crucial capability in real-world
applications such as video surveillance, meeting summarization, educational
lecture analysis, and sports broadcasting. However, it remains computationally
prohibitive for VideoLLMs, primarily due to two bottlenecks: 1) sequential
video decoding, the process of converting the raw bit stream to RGB frames can
take up to a minute for hour-long video inputs, and 2) costly prefilling of up
to several million tokens for LLM inference, resulting in high latency and
memory use. To address these challenges, we propose QuickVideo, a
system-algorithm co-design that substantially accelerates long-video
understanding to support real-time downstream applications. It comprises three
key innovations: QuickDecoder, a parallelized CPU-based video decoder that
achieves 2-3 times speedup by splitting videos into keyframe-aligned intervals
processed concurrently; QuickPrefill, a memory-efficient prefilling method
using KV-cache pruning to support more frames with less GPU memory; and an
overlapping scheme that overlaps CPU video decoding with GPU inference.
Together, these components infernece time reduce by a minute on long video
inputs, enabling scalable, high-quality video understanding even on limited
hardware. Experiments show that QuickVideo generalizes across durations and
sampling rates, making long video processing feasible in practice.
QuickVideo accelerates long-video understanding by combining a parallelized video decoder, memory-efficient prefilling, and overlapping video decoding with inference, enabling real-time performance.
QuickDecoder parallelized CPU-based video decoder keyframe-aligned intervals QuickPrefill memory-efficient prefilling KV-cache pruning overlapping scheme