title: Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformers
summary: State-of-the-art transformer-based large multimodal models (LMMs) struggle to
  handle hour-long video inputs due to the quadratic complexity of the causal
  self-attention operations, leading to high computational costs during training
  and inference. Existing token compression-based methods reduce the number of
  video tokens but often incur information loss and remain inefficient for
  extremely long sequences. In this paper, we explore an orthogonal direction to
  build a hybrid Mamba-Transformer model (VAMBA) that employs Mamba-2 blocks to
  encode video tokens with linear complexity. Without any token reduction, VAMBA
  can encode more than 1024 frames (640times360) on a single GPU, while
  transformer-based models can only encode 256 frames. On long video input, VAMBA
  achieves at least 50% reduction in GPU memory usage during training and
  inference, and nearly doubles the speed per training step compared to
  transformer-based LMMs. Our experimental results demonstrate that VAMBA
  improves accuracy by 4.3% on the challenging hour-long video understanding
  benchmark LVBench over prior efficient video LMMs, and maintains strong
  performance on a broad spectrum of long and short video understanding tasks.
paper_authors._id: 67d77743f918cc6703da2955
paper_authors.name: Weiming Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:49:43.554Z
paper_authors._id: 64405a9d518271b0d1beea38
paper_authors.avatarUrl: /avatars/b702474588fd7090773320422417a582.svg
paper_authors.fullname: Weiming Ren
paper_authors.type: user
paper_authors.user: wren93
paper_authors._id: 67d77743f918cc6703da2956
paper_authors.name: Wentao Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-17T16:33:51.694Z
paper_authors._id: 65c387c807a1445dfe1e9452
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65c387c807a1445dfe1e9452/t0VnwQh2wRZ9W_UGTZ8zt.jpeg
paper_authors.fullname: Wentao Ma
paper_authors.type: user
paper_authors.user: tonymwt
paper_authors._id: 67d77743f918cc6703da2957
paper_authors.name: Huan Yang
paper_authors._id: 67d77743f918cc6703da2958
paper_authors.name: Cong Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:50:04.030Z
paper_authors._id: 6470cc0a7fd7ecdbd0ec1482
paper_authors.avatarUrl: /avatars/1945ce9f933d199fc9ef17f12caebd2e.svg
paper_authors.fullname: Cong Wei
paper_authors.type: user
paper_authors.user: CongWei
paper_authors._id: 67d77743f918cc6703da2959
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:50:11.722Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 67d77743f918cc6703da295a
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:50:18.571Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_title: Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformers
paper_summary: State-of-the-art transformer-based large multimodal models (LMMs) struggle to
  handle hour-long video inputs due to the quadratic complexity of the causal
  self-attention operations, leading to high computational costs during training
  and inference. Existing token compression-based methods reduce the number of
  video tokens but often incur information loss and remain inefficient for
  extremely long sequences. In this paper, we explore an orthogonal direction to
  build a hybrid Mamba-Transformer model (VAMBA) that employs Mamba-2 blocks to
  encode video tokens with linear complexity. Without any token reduction, VAMBA
  can encode more than 1024 frames (640times360) on a single GPU, while
  transformer-based models can only encode 256 frames. On long video input, VAMBA
  achieves at least 50% reduction in GPU memory usage during training and
  inference, and nearly doubles the speed per training step compared to
  transformer-based LMMs. Our experimental results demonstrate that VAMBA
  improves accuracy by 4.3% on the challenging hour-long video understanding
  benchmark LVBench over prior efficient video LMMs, and maintains strong
  performance on a broad spectrum of long and short video understanding tasks.
paper_ai_summary: VAMBA, a hybrid Mamba-Transformer model, reduces computational costs and GPU memory usage while improving accuracy on long video inputs through linear complexity encoding.
paper_ai_keywords: transformer-based large multimodal models causal self-attention token compression Mamba-2 blocks VAMBA GPU memory usage training step speed hour-long video understanding LVBench video understanding tasks