title: EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State
    Space Duality
summary: For the deployment of neural networks in resource-constrained environments,
  prior works have built lightweight architectures with convolution and attention
  for capturing local and global dependencies, respectively. Recently, the state
  space model has emerged as an effective global token interaction with its
  favorable linear computational cost in the number of tokens. Yet, efficient
  vision backbones built with SSM have been explored less. In this paper, we
  introduce Efficient Vision Mamba (EfficientViM), a novel architecture built on
  hidden state mixer-based state space duality (HSM-SSD) that efficiently
  captures global dependencies with further reduced computational cost. In the
  HSM-SSD layer, we redesign the previous SSD layer to enable the channel mixing
  operation within hidden states. Additionally, we propose multi-stage hidden
  state fusion to further reinforce the representation power of hidden states,
  and provide the design alleviating the bottleneck caused by the memory-bound
  operations. As a result, the EfficientViM family achieves a new
  state-of-the-art speed-accuracy trade-off on ImageNet-1k, offering up to a 0.7%
  performance improvement over the second-best model SHViT with faster speed.
  Further, we observe significant improvements in throughput and accuracy
  compared to prior works, when scaling images or employing distillation
  training. Code is available at https://github.com/mlvlab/EfficientViM.
paper_authors._id: 6745a6cfad093f5ee502b9f1
paper_authors.name: Sanghyeok Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T09:20:02.295Z
paper_authors._id: 661748796b30e903469fac66
paper_authors.avatarUrl: /avatars/5cb8b0e9ee6c6cdfd350b42378e65ac7.svg
paper_authors.fullname: Sanghyeok Lee
paper_authors.type: user
paper_authors.user: SanghyeokLee
paper_authors._id: 6745a6cfad093f5ee502b9f2
paper_authors.name: Joonmyung Choi
paper_authors._id: 6745a6cfad093f5ee502b9f3
paper_authors.name: Hyunwoo J. Kim
paper_title: EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State
    Space Duality
paper_summary: For the deployment of neural networks in resource-constrained environments,
  prior works have built lightweight architectures with convolution and attention
  for capturing local and global dependencies, respectively. Recently, the state
  space model has emerged as an effective global token interaction with its
  favorable linear computational cost in the number of tokens. Yet, efficient
  vision backbones built with SSM have been explored less. In this paper, we
  introduce Efficient Vision Mamba (EfficientViM), a novel architecture built on
  hidden state mixer-based state space duality (HSM-SSD) that efficiently
  captures global dependencies with further reduced computational cost. In the
  HSM-SSD layer, we redesign the previous SSD layer to enable the channel mixing
  operation within hidden states. Additionally, we propose multi-stage hidden
  state fusion to further reinforce the representation power of hidden states,
  and provide the design alleviating the bottleneck caused by the memory-bound
  operations. As a result, the EfficientViM family achieves a new
  state-of-the-art speed-accuracy trade-off on ImageNet-1k, offering up to a 0.7%
  performance improvement over the second-best model SHViT with faster speed.
  Further, we observe significant improvements in throughput and accuracy
  compared to prior works, when scaling images or employing distillation
  training. Code is available at https://github.com/mlvlab/EfficientViM.
paper_ai_summary: Efficient Vision Mamba, using hidden state mixer-based state space duality, achieves improved speed-accuracy trade-offs in image classification and scaling.
paper_ai_keywords: state space model hidden state mixer state space duality EfficientViM multi-stage hidden state fusion ImageNet-1k SHViT distillation training