title: MambaMixer: Efficient Selective State Space Models with Dual Token and
    Channel Selection
summary: Recent advances in deep learning have mainly relied on Transformers due to
  their data dependency and ability to learn at scale. The attention module in
  these architectures, however, exhibits quadratic time and space in input size,
  limiting their scalability for long-sequence modeling. Despite recent attempts
  to design efficient and effective architecture backbone for multi-dimensional
  data, such as images and multivariate time series, existing models are either
  data independent, or fail to allow inter- and intra-dimension communication.
  Recently, State Space Models (SSMs), and more specifically Selective State
  Space Models, with efficient hardware-aware implementation, have shown
  promising potential for long sequence modeling. Motivated by the success of
  SSMs, we present MambaMixer, a new architecture with data-dependent weights
  that uses a dual selection mechanism across tokens and channels, called
  Selective Token and Channel Mixer. MambaMixer connects selective mixers using a
  weighted averaging mechanism, allowing layers to have direct access to early
  features. As a proof of concept, we design Vision MambaMixer (ViM2) and Time
  Series MambaMixer (TSM2) architectures based on the MambaMixer block and
  explore their performance in various vision and time series forecasting tasks.
  Our results underline the importance of selective mixing across both tokens and
  channels. In ImageNet classification, object detection, and semantic
  segmentation tasks, ViM2 achieves competitive performance with well-established
  vision models and outperforms SSM-based vision models. In time series
  forecasting, TSM2 achieves outstanding performance compared to state-of-the-art
  methods while demonstrating significantly improved computational cost. These
  results show that while Transformers, cross-channel attention, and MLPs are
  sufficient for good performance in time series forecasting, neither is
  necessary.
paper_authors._id: 660a407c4fa3a72a97b28d45
paper_authors.name: Ali Behrouz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-02T11:08:04.279Z
paper_authors._id: 65cccd5134a5d74cbaa9446c
paper_authors.avatarUrl: /avatars/5255b734628992106598eae4f2c5848f.svg
paper_authors.fullname: Ali Behrouz
paper_authors.type: user
paper_authors.user: AliBehrouz
paper_authors._id: 660a407c4fa3a72a97b28d46
paper_authors.name: Michele Santacatterina
paper_authors._id: 660a407c4fa3a72a97b28d47
paper_authors.name: Ramin Zabih
paper_title: MambaMixer: Efficient Selective State Space Models with Dual Token and
    Channel Selection
paper_summary: Recent advances in deep learning have mainly relied on Transformers due to
  their data dependency and ability to learn at scale. The attention module in
  these architectures, however, exhibits quadratic time and space in input size,
  limiting their scalability for long-sequence modeling. Despite recent attempts
  to design efficient and effective architecture backbone for multi-dimensional
  data, such as images and multivariate time series, existing models are either
  data independent, or fail to allow inter- and intra-dimension communication.
  Recently, State Space Models (SSMs), and more specifically Selective State
  Space Models, with efficient hardware-aware implementation, have shown
  promising potential for long sequence modeling. Motivated by the success of
  SSMs, we present MambaMixer, a new architecture with data-dependent weights
  that uses a dual selection mechanism across tokens and channels, called
  Selective Token and Channel Mixer. MambaMixer connects selective mixers using a
  weighted averaging mechanism, allowing layers to have direct access to early
  features. As a proof of concept, we design Vision MambaMixer (ViM2) and Time
  Series MambaMixer (TSM2) architectures based on the MambaMixer block and
  explore their performance in various vision and time series forecasting tasks.
  Our results underline the importance of selective mixing across both tokens and
  channels. In ImageNet classification, object detection, and semantic
  segmentation tasks, ViM2 achieves competitive performance with well-established
  vision models and outperforms SSM-based vision models. In time series
  forecasting, TSM2 achieves outstanding performance compared to state-of-the-art
  methods while demonstrating significantly improved computational cost. These
  results show that while Transformers, cross-channel attention, and MLPs are
  sufficient for good performance in time series forecasting, neither is
  necessary.
paper_ai_keywords: Transformers attention module data dependency data independent multi-dimensional data State Space Models (SSMs) Selective State Space Models data-dependent weights Selective Token and Channel Mixer weighted averaging mechanism early features Vision MambaMixer (ViM2) Time Series MambaMixer (TSM2) ImageNet classification object detection semantic segmentation time series forecasting computational cost cross-channel attention MLPs