title: Look Every Frame All at Once: Video-Ma^2mba for Efficient Long-form
    Video Understanding with Multi-Axis Gradient Checkpointing
summary: With the growing scale and complexity of video data, efficiently processing
  long video sequences poses significant challenges due to the quadratic increase
  in memory and computational demands associated with existing transformer-based
  Large Multi-modal Models (LMMs). To address these issues, we introduce
  Video-Ma^2mba, a novel architecture that incorporates State Space Models
  (SSMs) within the Mamba-2 framework, replacing the attention mechanisms. This
  allows the LMMs to scale linearly in terms of time and memory requirements,
  making it feasible to handle long-duration video content. Furthermore, we
  enhance the memory efficiency introducing the Multi-Axis Gradient Checkpointing
  (MA-GC) method, which strategically manages memory by retaining only essential
  activations across multiple computational axes. Our approach significantly
  reduces the memory footprint compared to standard gradient checkpointing.
  Empirical analyses show that Video-Ma^2mba can process extensive video
  sequences-equivalent to millions of tokens or over two hours of continuous
  sequences at 1 FPS-on a single GPU. By maintaining a detailed capture of
  temporal dynamics, our model improves the accuracy and relevance of responses
  in long video understanding tasks, demonstrating substantial advantages over
  existing frameworks.
paper_authors._id: 674d309e6a8ca5d12c0dac71
paper_authors.name: Hosu Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T08:49:36.175Z
paper_authors._id: 673060959e631f353ae1b5e0
paper_authors.avatarUrl: /avatars/d4b1e23de90ff1d02c38186a259b8d1e.svg
paper_authors.fullname: Hosu Lee
paper_authors.type: user
paper_authors.user: lakelee
paper_authors._id: 674d309e6a8ca5d12c0dac72
paper_authors.name: Junho Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T09:46:14.671Z
paper_authors._id: 653238bed0f5a9e537ed966d
paper_authors.avatarUrl: /avatars/e97a83e68e770baa1c5df847777cf213.svg
paper_authors.fullname: Junho Kim
paper_authors.type: user
paper_authors.user: arkimjh
paper_authors._id: 674d309e6a8ca5d12c0dac73
paper_authors.name: Hyunjun Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:16:15.390Z
paper_authors._id: 666419f8d8e1e34bc62166aa
paper_authors.avatarUrl: /avatars/9c419003788682800f0dced2f5b52f46.svg
paper_authors.fullname: Hyunjun Kim
paper_authors.type: user
paper_authors.user: kimhj709
paper_authors._id: 674d309e6a8ca5d12c0dac74
paper_authors.name: Yong Man Ro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:13:04.450Z
paper_authors._id: 660603529e3555d648e3baf1
paper_authors.avatarUrl: /avatars/0f54479afcfc19df00b25d5aedb4cf67.svg
paper_authors.fullname: Yong Man Ro
paper_authors.type: user
paper_authors.user: dwightro
paper_title: Look Every Frame All at Once: Video-Ma^2mba for Efficient Long-form
    Video Understanding with Multi-Axis Gradient Checkpointing
paper_summary: With the growing scale and complexity of video data, efficiently processing
  long video sequences poses significant challenges due to the quadratic increase
  in memory and computational demands associated with existing transformer-based
  Large Multi-modal Models (LMMs). To address these issues, we introduce
  Video-Ma^2mba, a novel architecture that incorporates State Space Models
  (SSMs) within the Mamba-2 framework, replacing the attention mechanisms. This
  allows the LMMs to scale linearly in terms of time and memory requirements,
  making it feasible to handle long-duration video content. Furthermore, we
  enhance the memory efficiency introducing the Multi-Axis Gradient Checkpointing
  (MA-GC) method, which strategically manages memory by retaining only essential
  activations across multiple computational axes. Our approach significantly
  reduces the memory footprint compared to standard gradient checkpointing.
  Empirical analyses show that Video-Ma^2mba can process extensive video
  sequences-equivalent to millions of tokens or over two hours of continuous
  sequences at 1 FPS-on a single GPU. By maintaining a detailed capture of
  temporal dynamics, our model improves the accuracy and relevance of responses
  in long video understanding tasks, demonstrating substantial advantages over
  existing frameworks.
paper_ai_summary: Video-Ma^2^mba, a novel architecture that replaces attention mechanisms with State Space Models in Mamba-2, efficiently processes long video sequences with linear memory and time scalability, enhancing memory efficiency with Multi-Axis Gradient Checkpointing.
paper_ai_keywords: State Space Models Mamba-2 Multi-Axis Gradient Checkpointing gradient checkpointing long video sequences