title: VSSD: Vision Mamba with Non-Casual State Space Duality
summary: Vision transformers have significantly advanced the field of computer vision,
  offering robust modeling capabilities and global receptive field. However,
  their high computational demands limit their applicability in processing long
  sequences. To tackle this issue, State Space Models (SSMs) have gained
  prominence in vision tasks as they offer linear computational complexity.
  Recently, State Space Duality (SSD), an improved variant of SSMs, was
  introduced in Mamba2 to enhance model performance and efficiency. However, the
  inherent causal nature of SSD/SSMs restricts their applications in non-causal
  vision tasks. To address this limitation, we introduce Visual State Space
  Duality (VSSD) model, which has a non-causal format of SSD. Specifically, we
  propose to discard the magnitude of interactions between the hidden state and
  tokens while preserving their relative weights, which relieves the dependencies
  of token contribution on previous tokens. Together with the involvement of
  multi-scan strategies, we show that the scanning results can be integrated to
  achieve non-causality, which not only improves the performance of SSD in vision
  tasks but also enhances its efficiency. We conduct extensive experiments on
  various benchmarks including image classification, detection, and segmentation,
  where VSSD surpasses existing state-of-the-art SSM-based models. Code and
  weights are available at https://github.com/YuHengsss/VSSD.
paper_authors._id: 66a6fc1de103701b1e53417f
paper_authors.name: Yuheng Shi
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-07-29T02:19:10.175Z
paper_authors._id: 65d6c252ca16ef9ba7e76664
paper_authors.avatarUrl: /avatars/45d6e8bdd7b944e912f6674696a941b3.svg
paper_authors.fullname: yuhengshi
paper_authors.type: user
paper_authors.user: YuhengSSS
paper_authors._id: 66a6fc1de103701b1e534180
paper_authors.name: Minjing Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-29T08:52:58.942Z
paper_authors._id: 645250dc1595002ddc8682d6
paper_authors.avatarUrl: /avatars/94d02ae79b2750a87c7e2a12a984531a.svg
paper_authors.fullname: Minjing DONG
paper_authors.type: user
paper_authors.user: UniSerj
paper_authors._id: 66a6fc1de103701b1e534181
paper_authors.name: Mingjia Li
paper_authors._id: 66a6fc1de103701b1e534182
paper_authors.name: Chang Xu
paper_title: VSSD: Vision Mamba with Non-Casual State Space Duality
paper_summary: Vision transformers have significantly advanced the field of computer vision,
  offering robust modeling capabilities and global receptive field. However,
  their high computational demands limit their applicability in processing long
  sequences. To tackle this issue, State Space Models (SSMs) have gained
  prominence in vision tasks as they offer linear computational complexity.
  Recently, State Space Duality (SSD), an improved variant of SSMs, was
  introduced in Mamba2 to enhance model performance and efficiency. However, the
  inherent causal nature of SSD/SSMs restricts their applications in non-causal
  vision tasks. To address this limitation, we introduce Visual State Space
  Duality (VSSD) model, which has a non-causal format of SSD. Specifically, we
  propose to discard the magnitude of interactions between the hidden state and
  tokens while preserving their relative weights, which relieves the dependencies
  of token contribution on previous tokens. Together with the involvement of
  multi-scan strategies, we show that the scanning results can be integrated to
  achieve non-causality, which not only improves the performance of SSD in vision
  tasks but also enhances its efficiency. We conduct extensive experiments on
  various benchmarks including image classification, detection, and segmentation,
  where VSSD surpasses existing state-of-the-art SSM-based models. Code and
  weights are available at https://github.com/YuHengsss/VSSD.
paper_ai_keywords: neural transformers global receptive field State Space Models (SSMs) State Space Duality (SSD) Visual State Space Duality (VSSD) multi-scan strategies