Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation
Multi-modal semantic segmentation significantly enhances AI agents'
perception and scene understanding, especially under adverse conditions like
low-light or overexposed environments. Leveraging additional modalities
(X-modality) like thermal and depth alongside traditional RGB provides
complementary information, enabling more robust and reliable segmentation. In
this work, we introduce Sigma, a Siamese Mamba network for multi-modal semantic
segmentation, utilizing the Selective Structured State Space Model, Mamba.
Unlike conventional methods that rely on CNNs, with their limited local
receptive fields, or Vision Transformers (ViTs), which offer global receptive
fields at the cost of quadratic complexity, our model achieves global receptive
fields coverage with linear complexity. By employing a Siamese encoder and
innovating a Mamba fusion mechanism, we effectively select essential
information from different modalities. A decoder is then developed to enhance
the channel-wise modeling ability of the model. Our method, Sigma, is
rigorously evaluated on both RGB-Thermal and RGB-Depth segmentation tasks,
demonstrating its superiority and marking the first successful application of
State Space Models (SSMs) in multi-modal perception tasks. Code is available at
https://github.com/zifuwan/Sigma.
6613bce7941ed67394314b18
Zifu Wan
claimed_verified
2025-05-27T15:50:58.640Z
65493c5b8fde27109badf365
/avatars/8ff7e84f7fe3cefd80c344b17986c5b3.svg
Zifu Wan
user
zifuwan
6613bce7941ed67394314b19
Yuhao Wang
6613bce7941ed67394314b1a
Silong Yong
6613bce7941ed67394314b1b
Pingping Zhang
6613bce7941ed67394314b1c
Simon Stepputtis
claimed_verified
2024-04-09T19:25:46.265Z
63214baffbf4f95480729d5f
https://cdn-avatars.huggingface.co/v1/production/uploads/63214baffbf4f95480729d5f/KyK3_wCq_pjyR5pEc5_Wc.png
Simon Stepputtis
user
sstepput
6613bce7941ed67394314b1d
Katia Sycara
6613bce7941ed67394314b1e
Yaqi Xie
claimed_verified
2024-04-09T19:25:52.857Z
661593183abccce37d1ece58
https://cdn-avatars.huggingface.co/v1/production/uploads/661593183abccce37d1ece58/uSQux3APhuLPYrgBABG3E.jpeg
Yaqi Xie
user
YaqiXie
Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation
Multi-modal semantic segmentation significantly enhances AI agents'
perception and scene understanding, especially under adverse conditions like
low-light or overexposed environments. Leveraging additional modalities
(X-modality) like thermal and depth alongside traditional RGB provides
complementary information, enabling more robust and reliable segmentation. In
this work, we introduce Sigma, a Siamese Mamba network for multi-modal semantic
segmentation, utilizing the Selective Structured State Space Model, Mamba.
Unlike conventional methods that rely on CNNs, with their limited local
receptive fields, or Vision Transformers (ViTs), which offer global receptive
fields at the cost of quadratic complexity, our model achieves global receptive
fields coverage with linear complexity. By employing a Siamese encoder and
innovating a Mamba fusion mechanism, we effectively select essential
information from different modalities. A decoder is then developed to enhance
the channel-wise modeling ability of the model. Our method, Sigma, is
rigorously evaluated on both RGB-Thermal and RGB-Depth segmentation tasks,
demonstrating its superiority and marking the first successful application of
State Space Models (SSMs) in multi-modal perception tasks. Code is available at
https://github.com/zifuwan/Sigma.
Sigma, a Siamese Mamba network using Selective Structured State Space Models, achieves efficient multi-modal semantic segmentation with linear complexity, outperforming CNNs and ViTs in adverse conditions.
Siamese Mamba network Selective Structured State Space Model Mamba fusion mechanism CNNs Vision Transformers global receptive fields linear complexity multi-modal semantic segmentation RGB-Thermal RGB-Depth State Space Models (SSMs)