VideoMamba: State Space Model for Efficient Video Understanding
Addressing the dual challenges of local redundancy and global dependencies in
video understanding, this work innovatively adapts the Mamba to the video
domain. The proposed VideoMamba overcomes the limitations of existing 3D
convolution neural networks and video transformers. Its linear-complexity
operator enables efficient long-term modeling, which is crucial for
high-resolution long video understanding. Extensive evaluations reveal
VideoMamba's four core abilities: (1) Scalability in the visual domain without
extensive dataset pretraining, thanks to a novel self-distillation technique;
(2) Sensitivity for recognizing short-term actions even with fine-grained
motion differences; (3) Superiority in long-term video understanding,
showcasing significant advancements over traditional feature-based models; and
(4) Compatibility with other modalities, demonstrating robustness in
multi-modal contexts. Through these distinct advantages, VideoMamba sets a new
benchmark for video understanding, offering a scalable and efficient solution
for comprehensive video understanding. All the code and models are available at
https://github.com/OpenGVLab/VideoMamba.
65efca60e2f3a2d6588c8360
Kunchang Li
claimed_verified
2024-03-12T12:06:04.510Z
61fb81006374891646732f37
https://cdn-avatars.huggingface.co/v1/production/uploads/1643872995181-61fb81006374891646732f37.jpeg
Kunchang Li
user
Andy1621
65efca60e2f3a2d6588c8361
Xinhao Li
claimed_verified
2025-05-26T08:48:46.432Z
64171b10f1e86908935a493a
https://cdn-avatars.huggingface.co/v1/production/uploads/64171b10f1e86908935a493a/tAwbHgTw714Vz-Qudz6Ij.jpeg
Xinhao Li
user
lixinhao
65efca60e2f3a2d6588c8362
Yi Wang
65efca60e2f3a2d6588c8363
Yinan He
claimed_verified
2024-03-18T08:36:40.805Z
62aafa49f29ff279b51f0182
https://cdn-avatars.huggingface.co/v1/production/uploads/62aafa49f29ff279b51f0182/rQx8QFQGOY2qIhqJ8zSRj.jpeg
yinanhe
user
ynhe
65efca60e2f3a2d6588c8364
Yali Wang
65efca60e2f3a2d6588c8365
Limin Wang
admin_assigned
2024-03-12T12:24:18.661Z
62c77f4352d8ae531f5511f9
/avatars/50198ccb02ccd286975a4613fbabee28.svg
Limin Wang
user
lmwang
65efca60e2f3a2d6588c8366
Yu Qiao
VideoMamba: State Space Model for Efficient Video Understanding
Addressing the dual challenges of local redundancy and global dependencies in
video understanding, this work innovatively adapts the Mamba to the video
domain. The proposed VideoMamba overcomes the limitations of existing 3D
convolution neural networks and video transformers. Its linear-complexity
operator enables efficient long-term modeling, which is crucial for
high-resolution long video understanding. Extensive evaluations reveal
VideoMamba's four core abilities: (1) Scalability in the visual domain without
extensive dataset pretraining, thanks to a novel self-distillation technique;
(2) Sensitivity for recognizing short-term actions even with fine-grained
motion differences; (3) Superiority in long-term video understanding,
showcasing significant advancements over traditional feature-based models; and
(4) Compatibility with other modalities, demonstrating robustness in
multi-modal contexts. Through these distinct advantages, VideoMamba sets a new
benchmark for video understanding, offering a scalable and efficient solution
for comprehensive video understanding. All the code and models are available at
https://github.com/OpenGVLab/VideoMamba.
VideoMamba, an adaptation of Mamba, addresses local and global challenges in video understanding through a linear-complexity operator, self-distillation, and compatibility with multiple modalities, setting a new benchmark.
Mamba VideoMamba 3D convolution neural networks video transformers linear-complexity operator self-distillation short-term actions long-term video understanding feature-based models multi-modal contexts