EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State
  Space Duality
For the deployment of neural networks in resource-constrained environments,
prior works have built lightweight architectures with convolution and attention
for capturing local and global dependencies, respectively. Recently, the state
space model has emerged as an effective global token interaction with its
favorable linear computational cost in the number of tokens. Yet, efficient
vision backbones built with SSM have been explored less. In this paper, we
introduce Efficient Vision Mamba (EfficientViM), a novel architecture built on
hidden state mixer-based state space duality (HSM-SSD) that efficiently
captures global dependencies with further reduced computational cost. In the
HSM-SSD layer, we redesign the previous SSD layer to enable the channel mixing
operation within hidden states. Additionally, we propose multi-stage hidden
state fusion to further reinforce the representation power of hidden states,
and provide the design alleviating the bottleneck caused by the memory-bound
operations. As a result, the EfficientViM family achieves a new
state-of-the-art speed-accuracy trade-off on ImageNet-1k, offering up to a 0.7%
performance improvement over the second-best model SHViT with faster speed.
Further, we observe significant improvements in throughput and accuracy
compared to prior works, when scaling images or employing distillation
training. Code is available at https://github.com/mlvlab/EfficientViM.
6745a6cfad093f5ee502b9f1
Sanghyeok Lee
claimed_verified
2024-11-28T09:20:02.295Z
661748796b30e903469fac66
/avatars/5cb8b0e9ee6c6cdfd350b42378e65ac7.svg
Sanghyeok Lee
user
SanghyeokLee
6745a6cfad093f5ee502b9f2
Joonmyung Choi
6745a6cfad093f5ee502b9f3
Hyunwoo J. Kim
claimed_verified
2025-06-16T15:51:29.123Z
63183a30a760ff8cff318be5
/avatars/1b314afd8d3319423915ffc2ae56d567.svg
Hyunwoo Kim
user
hkimcvml
EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State
  Space Duality
For the deployment of neural networks in resource-constrained environments,
prior works have built lightweight architectures with convolution and attention
for capturing local and global dependencies, respectively. Recently, the state
space model has emerged as an effective global token interaction with its
favorable linear computational cost in the number of tokens. Yet, efficient
vision backbones built with SSM have been explored less. In this paper, we
introduce Efficient Vision Mamba (EfficientViM), a novel architecture built on
hidden state mixer-based state space duality (HSM-SSD) that efficiently
captures global dependencies with further reduced computational cost. In the
HSM-SSD layer, we redesign the previous SSD layer to enable the channel mixing
operation within hidden states. Additionally, we propose multi-stage hidden
state fusion to further reinforce the representation power of hidden states,
and provide the design alleviating the bottleneck caused by the memory-bound
operations. As a result, the EfficientViM family achieves a new
state-of-the-art speed-accuracy trade-off on ImageNet-1k, offering up to a 0.7%
performance improvement over the second-best model SHViT with faster speed.
Further, we observe significant improvements in throughput and accuracy
compared to prior works, when scaling images or employing distillation
training. Code is available at https://github.com/mlvlab/EfficientViM.
Efficient Vision Mamba, using hidden state mixer-based state space duality, achieves improved speed-accuracy trade-offs in image classification and scaling.
state space model hidden state mixer state space duality EfficientViM multi-stage hidden state fusion ImageNet-1k SHViT distillation training