Video Mamba Suite: State Space Model as a Versatile Alternative for
  Video Understanding
Understanding videos is one of the fundamental directions in computer vision
research, with extensive efforts dedicated to exploring various architectures
such as RNN, 3D CNN, and Transformers. The newly proposed architecture of state
space model, e.g., Mamba, shows promising traits to extend its success in long
sequence modeling to video modeling. To assess whether Mamba can be a viable
alternative to Transformers in the video understanding domain, in this work, we
conduct a comprehensive set of studies, probing different roles Mamba can play
in modeling videos, while investigating diverse tasks where Mamba could exhibit
superiority. We categorize Mamba into four roles for modeling videos, deriving
a Video Mamba Suite composed of 14 models/modules, and evaluating them on 12
video understanding tasks. Our extensive experiments reveal the strong
potential of Mamba on both video-only and video-language tasks while showing
promising efficiency-performance trade-offs. We hope this work could provide
valuable data points and insights for future research on video understanding.
Code is public: https://github.com/OpenGVLab/video-mamba-suite.
65f3bffc57b928bdb17c5fd4
Guo Chen
claimed_verified
2024-03-19T08:46:11.507Z
6392c73390b8e99a6779a7b0
/avatars/9ff824ab02848120aec5e8de6780bcf1.svg
Guo Chen
user
cg1177
65f3bffc57b928bdb17c5fd5
Yifei Huang
65f3bffc57b928bdb17c5fd6
Jilan Xu
admin_assigned
2024-03-15T08:40:21.506Z
642e20d8018d9a57ca52af85
/avatars/c99c875a6654e7a8b3958777bcc46d8f.svg
Jilan Xu
user
Jazzcharles
65f3bffc57b928bdb17c5fd7
Baoqi Pei
65f3bffc57b928bdb17c5fd8
Zhe Chen
claimed_verified
2024-12-09T14:50:56.756Z
64119264f0f81eb569e0d569
/avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
Zhe Chen
user
czczup
65f3bffc57b928bdb17c5fd9
Zhiqi Li
claimed_verified
2025-01-28T10:27:29.372Z
6582d86e58df0a2e21db80b8
/avatars/a8245b1644183bd3ee7dc06b218c6e47.svg
ZhiqiLi
user
RealZhiqiLi
65f3bffc57b928bdb17c5fda
Jiahao Wang
65f3bffc57b928bdb17c5fdb
Kunchang Li
admin_assigned
2024-03-15T08:41:49.879Z
61fb81006374891646732f37
https://cdn-avatars.huggingface.co/v1/production/uploads/1643872995181-61fb81006374891646732f37.jpeg
Kunchang Li
user
Andy1621
65f3bffc57b928bdb17c5fdc
Tong Lu
65f3bffc57b928bdb17c5fdd
Limin Wang
admin_assigned
2024-03-15T08:42:32.957Z
62c77f4352d8ae531f5511f9
/avatars/50198ccb02ccd286975a4613fbabee28.svg
Limin Wang
user
lmwang
Video Mamba Suite: State Space Model as a Versatile Alternative for
  Video Understanding
Understanding videos is one of the fundamental directions in computer vision
research, with extensive efforts dedicated to exploring various architectures
such as RNN, 3D CNN, and Transformers. The newly proposed architecture of state
space model, e.g., Mamba, shows promising traits to extend its success in long
sequence modeling to video modeling. To assess whether Mamba can be a viable
alternative to Transformers in the video understanding domain, in this work, we
conduct a comprehensive set of studies, probing different roles Mamba can play
in modeling videos, while investigating diverse tasks where Mamba could exhibit
superiority. We categorize Mamba into four roles for modeling videos, deriving
a Video Mamba Suite composed of 14 models/modules, and evaluating them on 12
video understanding tasks. Our extensive experiments reveal the strong
potential of Mamba on both video-only and video-language tasks while showing
promising efficiency-performance trade-offs. We hope this work could provide
valuable data points and insights for future research on video understanding.
Code is public: https://github.com/OpenGVLab/video-mamba-suite.
The Video Mamba Suite demonstrates the strong potential of Mamba, a state space model, in video understanding tasks, showing its efficiency and performance advantages over Transformers.
state space model Mamba RNN 3D CNN Transformers video understanding video-only tasks video-language tasks