VMamba: Visual State Space Model
Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) stand as
the two most popular foundation models for visual representation learning.
While CNNs exhibit remarkable scalability with linear complexity w.r.t. image
resolution, ViTs surpass them in fitting capabilities despite contending with
quadratic complexity. A closer inspection reveals that ViTs achieve superior
visual modeling performance through the incorporation of global receptive
fields and dynamic weights. This observation motivates us to propose a novel
architecture that inherits these components while enhancing computational
efficiency. To this end, we draw inspiration from the recently introduced state
space model and propose the Visual State Space Model (VMamba), which achieves
linear complexity without sacrificing global receptive fields. To address the
encountered direction-sensitive issue, we introduce the Cross-Scan Module (CSM)
to traverse the spatial domain and convert any non-causal visual image into
order patch sequences. Extensive experimental results substantiate that VMamba
not only demonstrates promising capabilities across various visual perception
tasks, but also exhibits more pronounced advantages over established benchmarks
as the image resolution increases. Source code has been available at
https://github.com/MzeroMiko/VMamba.
65a9eb69cab1bfcc3afb8ff1
Yue Liu
65a9eb69cab1bfcc3afb8ff2
Yunjie Tian
admin_assigned
2024-01-19T13:44:43.598Z
646736d376bb704aa1401dbf
/avatars/302591dc0439eab4c9db2313b9bbf20a.svg
yunjie tian
user
sunsmarterjie
65a9eb69cab1bfcc3afb8ff3
Yuzhong Zhao
65a9eb69cab1bfcc3afb8ff4
Hongtian Yu
65a9eb69cab1bfcc3afb8ff5
Lingxi Xie
65a9eb69cab1bfcc3afb8ff6
Yaowei Wang
admin_assigned
2024-01-19T13:47:44.503Z
63e783b94577a86987a255bd
/avatars/6ea80c7430e412a70c9cb8b80398624b.svg
yaowei wang
user
yaoweiwang
65a9eb69cab1bfcc3afb8ff7
Qixiang Ye
65a9eb69cab1bfcc3afb8ff8
Yunfan Liu
VMamba: Visual State Space Model
Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) stand as
the two most popular foundation models for visual representation learning.
While CNNs exhibit remarkable scalability with linear complexity w.r.t. image
resolution, ViTs surpass them in fitting capabilities despite contending with
quadratic complexity. A closer inspection reveals that ViTs achieve superior
visual modeling performance through the incorporation of global receptive
fields and dynamic weights. This observation motivates us to propose a novel
architecture that inherits these components while enhancing computational
efficiency. To this end, we draw inspiration from the recently introduced state
space model and propose the Visual State Space Model (VMamba), which achieves
linear complexity without sacrificing global receptive fields. To address the
encountered direction-sensitive issue, we introduce the Cross-Scan Module (CSM)
to traverse the spatial domain and convert any non-causal visual image into
order patch sequences. Extensive experimental results substantiate that VMamba
not only demonstrates promising capabilities across various visual perception
tasks, but also exhibits more pronounced advantages over established benchmarks
as the image resolution increases. Source code has been available at
https://github.com/MzeroMiko/VMamba.
VMamba, a novel state space model architecture, combines global receptive fields and dynamic weights from ViTs with linear complexity, outperforming established models as image resolution increases.
Convolutional Neural Networks Vision Transformers global receptive fields dynamic weights state space model Visual State Space Model VMamba linear complexity Cross-Scan Module CSM spatial domain order patch sequences