title: Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation
summary: Multi-modal semantic segmentation significantly enhances AI agents'
  perception and scene understanding, especially under adverse conditions like
  low-light or overexposed environments. Leveraging additional modalities
  (X-modality) like thermal and depth alongside traditional RGB provides
  complementary information, enabling more robust and reliable segmentation. In
  this work, we introduce Sigma, a Siamese Mamba network for multi-modal semantic
  segmentation, utilizing the Selective Structured State Space Model, Mamba.
  Unlike conventional methods that rely on CNNs, with their limited local
  receptive fields, or Vision Transformers (ViTs), which offer global receptive
  fields at the cost of quadratic complexity, our model achieves global receptive
  fields coverage with linear complexity. By employing a Siamese encoder and
  innovating a Mamba fusion mechanism, we effectively select essential
  information from different modalities. A decoder is then developed to enhance
  the channel-wise modeling ability of the model. Our method, Sigma, is
  rigorously evaluated on both RGB-Thermal and RGB-Depth segmentation tasks,
  demonstrating its superiority and marking the first successful application of
  State Space Models (SSMs) in multi-modal perception tasks. Code is available at
  https://github.com/zifuwan/Sigma.
paper_authors._id: 6613bce7941ed67394314b18
paper_authors.name: Zifu Wan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T15:50:58.640Z
paper_authors._id: 65493c5b8fde27109badf365
paper_authors.avatarUrl: /avatars/8ff7e84f7fe3cefd80c344b17986c5b3.svg
paper_authors.fullname: Zifu Wan
paper_authors.type: user
paper_authors.user: zifuwan
paper_authors._id: 6613bce7941ed67394314b19
paper_authors.name: Yuhao Wang
paper_authors._id: 6613bce7941ed67394314b1a
paper_authors.name: Silong Yong
paper_authors._id: 6613bce7941ed67394314b1b
paper_authors.name: Pingping Zhang
paper_authors._id: 6613bce7941ed67394314b1c
paper_authors.name: Simon Stepputtis
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-09T19:25:46.265Z
paper_authors._id: 63214baffbf4f95480729d5f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63214baffbf4f95480729d5f/KyK3_wCq_pjyR5pEc5_Wc.png
paper_authors.fullname: Simon Stepputtis
paper_authors.type: user
paper_authors.user: sstepput
paper_authors._id: 6613bce7941ed67394314b1d
paper_authors.name: Katia Sycara
paper_authors._id: 6613bce7941ed67394314b1e
paper_authors.name: Yaqi Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-09T19:25:52.857Z
paper_authors._id: 661593183abccce37d1ece58
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/661593183abccce37d1ece58/uSQux3APhuLPYrgBABG3E.jpeg
paper_authors.fullname: Yaqi Xie
paper_authors.type: user
paper_authors.user: YaqiXie
paper_title: Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation
paper_summary: Multi-modal semantic segmentation significantly enhances AI agents'
  perception and scene understanding, especially under adverse conditions like
  low-light or overexposed environments. Leveraging additional modalities
  (X-modality) like thermal and depth alongside traditional RGB provides
  complementary information, enabling more robust and reliable segmentation. In
  this work, we introduce Sigma, a Siamese Mamba network for multi-modal semantic
  segmentation, utilizing the Selective Structured State Space Model, Mamba.
  Unlike conventional methods that rely on CNNs, with their limited local
  receptive fields, or Vision Transformers (ViTs), which offer global receptive
  fields at the cost of quadratic complexity, our model achieves global receptive
  fields coverage with linear complexity. By employing a Siamese encoder and
  innovating a Mamba fusion mechanism, we effectively select essential
  information from different modalities. A decoder is then developed to enhance
  the channel-wise modeling ability of the model. Our method, Sigma, is
  rigorously evaluated on both RGB-Thermal and RGB-Depth segmentation tasks,
  demonstrating its superiority and marking the first successful application of
  State Space Models (SSMs) in multi-modal perception tasks. Code is available at
  https://github.com/zifuwan/Sigma.
paper_ai_summary: Sigma, a Siamese Mamba network using Selective Structured State Space Models, achieves efficient multi-modal semantic segmentation with linear complexity, outperforming CNNs and ViTs in adverse conditions.
paper_ai_keywords: Siamese Mamba network Selective Structured State Space Model Mamba fusion mechanism CNNs Vision Transformers global receptive fields linear complexity multi-modal semantic segmentation RGB-Thermal RGB-Depth State Space Models (SSMs)