title: LocalMamba: Visual State Space Model with Windowed Selective Scan
summary: Recent advancements in state space models, notably Mamba, have demonstrated
  significant progress in modeling long sequences for tasks like language
  understanding. Yet, their application in vision tasks has not markedly
  surpassed the performance of traditional Convolutional Neural Networks (CNNs)
  and Vision Transformers (ViTs). This paper posits that the key to enhancing
  Vision Mamba (ViM) lies in optimizing scan directions for sequence modeling.
  Traditional ViM approaches, which flatten spatial tokens, overlook the
  preservation of local 2D dependencies, thereby elongating the distance between
  adjacent tokens. We introduce a novel local scanning strategy that divides
  images into distinct windows, effectively capturing local dependencies while
  maintaining a global perspective. Additionally, acknowledging the varying
  preferences for scan patterns across different network layers, we propose a
  dynamic method to independently search for the optimal scan choices for each
  layer, substantially improving performance. Extensive experiments across both
  plain and hierarchical models underscore our approach's superiority in
  effectively capturing image representations. For example, our model
  significantly outperforms Vim-Ti by 3.1% on ImageNet with the same 1.5G FLOPs.
  Code is available at: https://github.com/hunto/LocalMamba.
paper_authors._id: 65f3bc49f5b6c6b52fc08c29
paper_authors.name: Tao Huang
paper_authors._id: 65f3bc49f5b6c6b52fc08c2a
paper_authors.name: Xiaohuan Pei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-16T00:02:44.701Z
paper_authors._id: 6318661051edb569f1b4e68b
paper_authors.avatarUrl: /avatars/9b98943bc609ca974abf0abff3e68633.svg
paper_authors.fullname: TerryPei
paper_authors.type: user
paper_authors.user: TerryPei
paper_authors._id: 65f3bc49f5b6c6b52fc08c2b
paper_authors.name: Shan You
paper_authors._id: 65f3bc49f5b6c6b52fc08c2c
paper_authors.name: Fei Wang
paper_authors._id: 65f3bc49f5b6c6b52fc08c2d
paper_authors.name: Chen Qian
paper_authors._id: 65f3bc49f5b6c6b52fc08c2e
paper_authors.name: Chang Xu
paper_title: LocalMamba: Visual State Space Model with Windowed Selective Scan
paper_summary: Recent advancements in state space models, notably Mamba, have demonstrated
  significant progress in modeling long sequences for tasks like language
  understanding. Yet, their application in vision tasks has not markedly
  surpassed the performance of traditional Convolutional Neural Networks (CNNs)
  and Vision Transformers (ViTs). This paper posits that the key to enhancing
  Vision Mamba (ViM) lies in optimizing scan directions for sequence modeling.
  Traditional ViM approaches, which flatten spatial tokens, overlook the
  preservation of local 2D dependencies, thereby elongating the distance between
  adjacent tokens. We introduce a novel local scanning strategy that divides
  images into distinct windows, effectively capturing local dependencies while
  maintaining a global perspective. Additionally, acknowledging the varying
  preferences for scan patterns across different network layers, we propose a
  dynamic method to independently search for the optimal scan choices for each
  layer, substantially improving performance. Extensive experiments across both
  plain and hierarchical models underscore our approach's superiority in
  effectively capturing image representations. For example, our model
  significantly outperforms Vim-Ti by 3.1% on ImageNet with the same 1.5G FLOPs.
  Code is available at: https://github.com/hunto/LocalMamba.
paper_ai_summary: A novel local scanning strategy for Vision Mamba improves vision task performance by optimizing scan directions and maintaining local 2D dependencies across distinct layers.
paper_ai_keywords: state space models Mamba Vision Mamba sequence modeling spatial tokens local dependencies scan directions image windows dynamic scanning ImageNet FLOPs