title: Video Mamba Suite: State Space Model as a Versatile Alternative for
    Video Understanding
summary: Understanding videos is one of the fundamental directions in computer vision
  research, with extensive efforts dedicated to exploring various architectures
  such as RNN, 3D CNN, and Transformers. The newly proposed architecture of state
  space model, e.g., Mamba, shows promising traits to extend its success in long
  sequence modeling to video modeling. To assess whether Mamba can be a viable
  alternative to Transformers in the video understanding domain, in this work, we
  conduct a comprehensive set of studies, probing different roles Mamba can play
  in modeling videos, while investigating diverse tasks where Mamba could exhibit
  superiority. We categorize Mamba into four roles for modeling videos, deriving
  a Video Mamba Suite composed of 14 models/modules, and evaluating them on 12
  video understanding tasks. Our extensive experiments reveal the strong
  potential of Mamba on both video-only and video-language tasks while showing
  promising efficiency-performance trade-offs. We hope this work could provide
  valuable data points and insights for future research on video understanding.
  Code is public: https://github.com/OpenGVLab/video-mamba-suite.
paper_authors._id: 65f3bffc57b928bdb17c5fd4
paper_authors.name: Guo Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-19T08:46:11.507Z
paper_authors._id: 6392c73390b8e99a6779a7b0
paper_authors.avatarUrl: /avatars/9ff824ab02848120aec5e8de6780bcf1.svg
paper_authors.fullname: Guo Chen
paper_authors.type: user
paper_authors.user: cg1177
paper_authors._id: 65f3bffc57b928bdb17c5fd5
paper_authors.name: Yifei Huang
paper_authors._id: 65f3bffc57b928bdb17c5fd6
paper_authors.name: Jilan Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:40:21.506Z
paper_authors._id: 642e20d8018d9a57ca52af85
paper_authors.avatarUrl: /avatars/c99c875a6654e7a8b3958777bcc46d8f.svg
paper_authors.fullname: Jilan Xu
paper_authors.type: user
paper_authors.user: Jazzcharles
paper_authors._id: 65f3bffc57b928bdb17c5fd7
paper_authors.name: Baoqi Pei
paper_authors._id: 65f3bffc57b928bdb17c5fd8
paper_authors.name: Zhe Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T14:50:56.756Z
paper_authors._id: 64119264f0f81eb569e0d569
paper_authors.avatarUrl: /avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
paper_authors.fullname: Zhe Chen
paper_authors.type: user
paper_authors.user: czczup
paper_authors._id: 65f3bffc57b928bdb17c5fd9
paper_authors.name: Zhiqi Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-28T10:27:29.372Z
paper_authors._id: 6582d86e58df0a2e21db80b8
paper_authors.avatarUrl: /avatars/a8245b1644183bd3ee7dc06b218c6e47.svg
paper_authors.fullname: ZhiqiLi
paper_authors.type: user
paper_authors.user: RealZhiqiLi
paper_authors._id: 65f3bffc57b928bdb17c5fda
paper_authors.name: Jiahao Wang
paper_authors._id: 65f3bffc57b928bdb17c5fdb
paper_authors.name: Kunchang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:41:49.879Z
paper_authors._id: 61fb81006374891646732f37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1643872995181-61fb81006374891646732f37.jpeg
paper_authors.fullname: Kunchang Li
paper_authors.type: user
paper_authors.user: Andy1621
paper_authors._id: 65f3bffc57b928bdb17c5fdc
paper_authors.name: Tong Lu
paper_authors._id: 65f3bffc57b928bdb17c5fdd
paper_authors.name: Limin Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:42:32.957Z
paper_authors._id: 62c77f4352d8ae531f5511f9
paper_authors.avatarUrl: /avatars/50198ccb02ccd286975a4613fbabee28.svg
paper_authors.fullname: Limin Wang
paper_authors.type: user
paper_authors.user: lmwang
paper_title: Video Mamba Suite: State Space Model as a Versatile Alternative for
    Video Understanding
paper_summary: Understanding videos is one of the fundamental directions in computer vision
  research, with extensive efforts dedicated to exploring various architectures
  such as RNN, 3D CNN, and Transformers. The newly proposed architecture of state
  space model, e.g., Mamba, shows promising traits to extend its success in long
  sequence modeling to video modeling. To assess whether Mamba can be a viable
  alternative to Transformers in the video understanding domain, in this work, we
  conduct a comprehensive set of studies, probing different roles Mamba can play
  in modeling videos, while investigating diverse tasks where Mamba could exhibit
  superiority. We categorize Mamba into four roles for modeling videos, deriving
  a Video Mamba Suite composed of 14 models/modules, and evaluating them on 12
  video understanding tasks. Our extensive experiments reveal the strong
  potential of Mamba on both video-only and video-language tasks while showing
  promising efficiency-performance trade-offs. We hope this work could provide
  valuable data points and insights for future research on video understanding.
  Code is public: https://github.com/OpenGVLab/video-mamba-suite.
paper_ai_keywords: RNN 3D CNN Transformers state space model Mamba long sequence modeling video modeling Video Mamba Suite video-only tasks video-language tasks efficiency-performance trade-offs