MambaVision: A Hybrid Mamba-Transformer Vision Backbone
We propose a novel hybrid Mamba-Transformer backbone, denoted as MambaVision,
which is specifically tailored for vision applications. Our core contribution
includes redesigning the Mamba formulation to enhance its capability for
efficient modeling of visual features. In addition, we conduct a comprehensive
ablation study on the feasibility of integrating Vision Transformers (ViT) with
Mamba. Our results demonstrate that equipping the Mamba architecture with
several self-attention blocks at the final layers greatly improves the modeling
capacity to capture long-range spatial dependencies. Based on our findings, we
introduce a family of MambaVision models with a hierarchical architecture to
meet various design criteria. For Image classification on ImageNet-1K dataset,
MambaVision model variants achieve a new State-of-the-Art (SOTA) performance in
terms of Top-1 accuracy and image throughput. In downstream tasks such as
object detection, instance segmentation and semantic segmentation on MS COCO
and ADE20K datasets, MambaVision outperforms comparably-sized backbones and
demonstrates more favorable performance. Code:
https://github.com/NVlabs/MambaVision.
66908c6ab70d356ed3bd357e
Ali Hatamizadeh
claimed_verified
2024-07-12T16:03:28.880Z
64414b62603214724ebd2636
https://cdn-avatars.huggingface.co/v1/production/uploads/64414b62603214724ebd2636/x9JVcJRZKZE7hdEII1JRR.jpeg
Ali
user
ahatamiz
66908c6ab70d356ed3bd357f
Jan Kautz
MambaVision: A Hybrid Mamba-Transformer Vision Backbone
We propose a novel hybrid Mamba-Transformer backbone, denoted as MambaVision,
which is specifically tailored for vision applications. Our core contribution
includes redesigning the Mamba formulation to enhance its capability for
efficient modeling of visual features. In addition, we conduct a comprehensive
ablation study on the feasibility of integrating Vision Transformers (ViT) with
Mamba. Our results demonstrate that equipping the Mamba architecture with
several self-attention blocks at the final layers greatly improves the modeling
capacity to capture long-range spatial dependencies. Based on our findings, we
introduce a family of MambaVision models with a hierarchical architecture to
meet various design criteria. For Image classification on ImageNet-1K dataset,
MambaVision model variants achieve a new State-of-the-Art (SOTA) performance in
terms of Top-1 accuracy and image throughput. In downstream tasks such as
object detection, instance segmentation and semantic segmentation on MS COCO
and ADE20K datasets, MambaVision outperforms comparably-sized backbones and
demonstrates more favorable performance. Code:
https://github.com/NVlabs/MambaVision.
MambaVision, a hybrid Mamba-Transformer backbone, enhances visual feature modeling with self-attention blocks, achieving state-of-the-art performance in image classification and downstream tasks.
Mamba-Transformer MambaVision self-attention long-range spatial dependencies hierarchical architecture ImageNet-1K Top-1 accuracy image throughput object detection instance segmentation semantic segmentation MS COCO ADE20K