title: MambaVision: A Hybrid Mamba-Transformer Vision Backbone
summary: We propose a novel hybrid Mamba-Transformer backbone, denoted as MambaVision,
  which is specifically tailored for vision applications. Our core contribution
  includes redesigning the Mamba formulation to enhance its capability for
  efficient modeling of visual features. In addition, we conduct a comprehensive
  ablation study on the feasibility of integrating Vision Transformers (ViT) with
  Mamba. Our results demonstrate that equipping the Mamba architecture with
  several self-attention blocks at the final layers greatly improves the modeling
  capacity to capture long-range spatial dependencies. Based on our findings, we
  introduce a family of MambaVision models with a hierarchical architecture to
  meet various design criteria. For Image classification on ImageNet-1K dataset,
  MambaVision model variants achieve a new State-of-the-Art (SOTA) performance in
  terms of Top-1 accuracy and image throughput. In downstream tasks such as
  object detection, instance segmentation and semantic segmentation on MS COCO
  and ADE20K datasets, MambaVision outperforms comparably-sized backbones and
  demonstrates more favorable performance. Code:
  https://github.com/NVlabs/MambaVision.
paper_authors._id: 66908c6ab70d356ed3bd357e
paper_authors.name: Ali Hatamizadeh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T16:03:28.880Z
paper_authors._id: 64414b62603214724ebd2636
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64414b62603214724ebd2636/x9JVcJRZKZE7hdEII1JRR.jpeg
paper_authors.fullname: Ali
paper_authors.type: user
paper_authors.user: ahatamiz
paper_authors._id: 66908c6ab70d356ed3bd357f
paper_authors.name: Jan Kautz
paper_title: MambaVision: A Hybrid Mamba-Transformer Vision Backbone
paper_summary: We propose a novel hybrid Mamba-Transformer backbone, denoted as MambaVision,
  which is specifically tailored for vision applications. Our core contribution
  includes redesigning the Mamba formulation to enhance its capability for
  efficient modeling of visual features. In addition, we conduct a comprehensive
  ablation study on the feasibility of integrating Vision Transformers (ViT) with
  Mamba. Our results demonstrate that equipping the Mamba architecture with
  several self-attention blocks at the final layers greatly improves the modeling
  capacity to capture long-range spatial dependencies. Based on our findings, we
  introduce a family of MambaVision models with a hierarchical architecture to
  meet various design criteria. For Image classification on ImageNet-1K dataset,
  MambaVision model variants achieve a new State-of-the-Art (SOTA) performance in
  terms of Top-1 accuracy and image throughput. In downstream tasks such as
  object detection, instance segmentation and semantic segmentation on MS COCO
  and ADE20K datasets, MambaVision outperforms comparably-sized backbones and
  demonstrates more favorable performance. Code:
  https://github.com/NVlabs/MambaVision.
paper_ai_keywords: Mamba-Transformer MambaVision Vision Transformers (ViT) self-attention blocks long-range spatial dependencies hierarchical architecture Image classification ImageNet-1K Top-1 accuracy image throughput object detection instance segmentation semantic segmentation MS COCO ADE2K