title: Vision Mamba: Efficient Visual Representation Learning with
    Bidirectional State Space Model
summary: Recently the state space models (SSMs) with efficient hardware-aware designs,
  i.e., Mamba, have shown great potential for long sequence modeling. Building
  efficient and generic vision backbones purely upon SSMs is an appealing
  direction. However, representing visual data is challenging for SSMs due to the
  position-sensitivity of visual data and the requirement of global context for
  visual understanding. In this paper, we show that the reliance of visual
  representation learning on self-attention is not necessary and propose a new
  generic vision backbone with bidirectional Mamba blocks (Vim), which marks the
  image sequences with position embeddings and compresses the visual
  representation with bidirectional state space models. On ImageNet
  classification, COCO object detection, and ADE20k semantic segmentation tasks,
  Vim achieves higher performance compared to well-established vision
  transformers like DeiT, while also demonstrating significantly improved
  computation & memory efficiency. For example, Vim is 2.8times faster than
  DeiT and saves 86.8% GPU memory when performing batch inference to extract
  features on images with a resolution of 1248times1248. The results
  demonstrate that Vim is capable of overcoming the computation & memory
  constraints on performing Transformer-style understanding for high-resolution
  images and it has great potential to become the next-generation backbone for
  vision foundation models. Code is available at https://github.com/hustvl/Vim.
paper_authors._id: 65a8aa9e87dd2bd7f900d536
paper_authors.name: Lianghui Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T09:03:35.982Z
paper_authors._id: 643632c6a4bd75c62cc2d3b4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643632c6a4bd75c62cc2d3b4/rCqEb3_w54-QH2QSP2JhM.jpeg
paper_authors.fullname: Lianghui Zhu
paper_authors.type: user
paper_authors.user: LianghuiZhu
paper_authors._id: 65a8aa9e87dd2bd7f900d537
paper_authors.name: Bencheng Liao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-27T22:21:19.701Z
paper_authors._id: 6577073fc2bf55b1f6bafb49
paper_authors.avatarUrl: /avatars/58803398b1a918b7570db17893e65122.svg
paper_authors.fullname: liao
paper_authors.type: user
paper_authors.user: LegendBC
paper_authors._id: 65a8aa9e87dd2bd7f900d538
paper_authors.name: Qian Zhang
paper_authors._id: 65a8aa9e87dd2bd7f900d539
paper_authors.name: Xinlong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T09:04:16.206Z
paper_authors._id: 63ca558304c979828311c5a5
paper_authors.avatarUrl: /avatars/2a439d79fba2f987cabe780d10c94d25.svg
paper_authors.fullname: Xinlong Wang
paper_authors.type: user
paper_authors.user: xinlongwang
paper_authors._id: 65a8aa9e87dd2bd7f900d53a
paper_authors.name: Wenyu Liu
paper_authors._id: 65a8aa9e87dd2bd7f900d53b
paper_authors.name: Xinggang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T09:04:50.536Z
paper_authors._id: 62600de6d47e3dbae32ce1ce
paper_authors.avatarUrl: /avatars/a536417cfec6e10ac415091bd1829426.svg
paper_authors.fullname: Xinggang Wang
paper_authors.type: user
paper_authors.user: xinggangw
paper_title: Vision Mamba: Efficient Visual Representation Learning with
    Bidirectional State Space Model
paper_summary: Recently the state space models (SSMs) with efficient hardware-aware designs,
  i.e., Mamba, have shown great potential for long sequence modeling. Building
  efficient and generic vision backbones purely upon SSMs is an appealing
  direction. However, representing visual data is challenging for SSMs due to the
  position-sensitivity of visual data and the requirement of global context for
  visual understanding. In this paper, we show that the reliance of visual
  representation learning on self-attention is not necessary and propose a new
  generic vision backbone with bidirectional Mamba blocks (Vim), which marks the
  image sequences with position embeddings and compresses the visual
  representation with bidirectional state space models. On ImageNet
  classification, COCO object detection, and ADE20k semantic segmentation tasks,
  Vim achieves higher performance compared to well-established vision
  transformers like DeiT, while also demonstrating significantly improved
  computation & memory efficiency. For example, Vim is 2.8times faster than
  DeiT and saves 86.8% GPU memory when performing batch inference to extract
  features on images with a resolution of 1248times1248. The results
  demonstrate that Vim is capable of overcoming the computation & memory
  constraints on performing Transformer-style understanding for high-resolution
  images and it has great potential to become the next-generation backbone for
  vision foundation models. Code is available at https://github.com/hustvl/Vim.
paper_ai_keywords: state space models (SSMs) Mamba bidirectional Mamba blocks (Vim) position embeddings bidirectional state space models ImageNet classification COCO object detection ADE20k semantic segmentation vision transformers DeiT inference efficiency memory efficiency high-resolution images vision foundation models