Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformers
State-of-the-art transformer-based large multimodal models (LMMs) struggle to
handle hour-long video inputs due to the quadratic complexity of the causal
self-attention operations, leading to high computational costs during training
and inference. Existing token compression-based methods reduce the number of
video tokens but often incur information loss and remain inefficient for
extremely long sequences. In this paper, we explore an orthogonal direction to
build a hybrid Mamba-Transformer model (VAMBA) that employs Mamba-2 blocks to
encode video tokens with linear complexity. Without any token reduction, VAMBA
can encode more than 1024 frames (640times360) on a single GPU, while
transformer-based models can only encode 256 frames. On long video input, VAMBA
achieves at least 50% reduction in GPU memory usage during training and
inference, and nearly doubles the speed per training step compared to
transformer-based LMMs. Our experimental results demonstrate that VAMBA
improves accuracy by 4.3% on the challenging hour-long video understanding
benchmark LVBench over prior efficient video LMMs, and maintains strong
performance on a broad spectrum of long and short video understanding tasks.
67d77743f918cc6703da2955
Weiming Ren
admin_assigned
2025-03-17T14:49:43.554Z
64405a9d518271b0d1beea38
/avatars/b702474588fd7090773320422417a582.svg
Weiming Ren
user
wren93
67d77743f918cc6703da2956
Wentao Ma
claimed_verified
2025-03-17T16:33:51.694Z
65c387c807a1445dfe1e9452
https://cdn-avatars.huggingface.co/v1/production/uploads/65c387c807a1445dfe1e9452/NCsYIYrCQxQcPBiMe_4ZI.jpeg
Wentao Ma
user
tonymwt
67d77743f918cc6703da2957
Huan Yang
67d77743f918cc6703da2958
Cong Wei
admin_assigned
2025-03-17T14:50:04.030Z
6470cc0a7fd7ecdbd0ec1482
/avatars/1945ce9f933d199fc9ef17f12caebd2e.svg
Cong Wei
user
CongWei
67d77743f918cc6703da2959
Ge Zhang
admin_assigned
2025-03-17T14:50:11.722Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
67d77743f918cc6703da295a
Wenhu Chen
admin_assigned
2025-03-17T14:50:18.571Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformers
State-of-the-art transformer-based large multimodal models (LMMs) struggle to
handle hour-long video inputs due to the quadratic complexity of the causal
self-attention operations, leading to high computational costs during training
and inference. Existing token compression-based methods reduce the number of
video tokens but often incur information loss and remain inefficient for
extremely long sequences. In this paper, we explore an orthogonal direction to
build a hybrid Mamba-Transformer model (VAMBA) that employs Mamba-2 blocks to
encode video tokens with linear complexity. Without any token reduction, VAMBA
can encode more than 1024 frames (640times360) on a single GPU, while
transformer-based models can only encode 256 frames. On long video input, VAMBA
achieves at least 50% reduction in GPU memory usage during training and
inference, and nearly doubles the speed per training step compared to
transformer-based LMMs. Our experimental results demonstrate that VAMBA
improves accuracy by 4.3% on the challenging hour-long video understanding
benchmark LVBench over prior efficient video LMMs, and maintains strong
performance on a broad spectrum of long and short video understanding tasks.
VAMBA, a hybrid Mamba-Transformer model, reduces computational costs and GPU memory usage while improving accuracy on long video inputs through linear complexity encoding.
transformer-based large multimodal models causal self-attention token compression Mamba-2 blocks VAMBA GPU memory usage training step speed hour-long video understanding LVBench video understanding tasks