title: SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate
    Time series
summary: Transformers have widely adopted attention networks for sequence mixing and
  MLPs for channel mixing, playing a pivotal role in achieving breakthroughs
  across domains. However, recent literature highlights issues with attention
  networks, including low inductive bias and quadratic complexity concerning
  input sequence length. State Space Models (SSMs) like S4 and others (Hippo,
  Global Convolutions, liquid S4, LRU, Mega, and Mamba), have emerged to address
  the above issues to help handle longer sequence lengths. Mamba, while being the
  state-of-the-art SSM, has a stability issue when scaled to large networks for
  computer vision datasets. We propose SiMBA, a new architecture that introduces
  Einstein FFT (EinFFT) for channel modeling by specific eigenvalue computations
  and uses the Mamba block for sequence modeling. Extensive performance studies
  across image and time-series benchmarks demonstrate that SiMBA outperforms
  existing SSMs, bridging the performance gap with state-of-the-art transformers.
  Notably, SiMBA establishes itself as the new state-of-the-art SSM on ImageNet
  and transfer learning benchmarks such as Stanford Car and Flower as well as
  task learning benchmarks as well as seven time series benchmark datasets. The
  project page is available on this website
  ~https://github.com/badripatro/Simba.
paper_authors._id: 6600d54e4d7ed9ef948b1277
paper_authors.name: Badri N. Patro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:50:15.761Z
paper_authors._id: 64abec030ba6725d692bcd0f
paper_authors.avatarUrl: /avatars/8a5cc786de279726bdf6b194f21ac3cb.svg
paper_authors.fullname: Badri Narayana Patro
paper_authors.type: user
paper_authors.user: badripatro
paper_authors._id: 6600d54e4d7ed9ef948b1278
paper_authors.name: Vijay S. Agneeswaran
paper_title: SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate
    Time series
paper_summary: Transformers have widely adopted attention networks for sequence mixing and
  MLPs for channel mixing, playing a pivotal role in achieving breakthroughs
  across domains. However, recent literature highlights issues with attention
  networks, including low inductive bias and quadratic complexity concerning
  input sequence length. State Space Models (SSMs) like S4 and others (Hippo,
  Global Convolutions, liquid S4, LRU, Mega, and Mamba), have emerged to address
  the above issues to help handle longer sequence lengths. Mamba, while being the
  state-of-the-art SSM, has a stability issue when scaled to large networks for
  computer vision datasets. We propose SiMBA, a new architecture that introduces
  Einstein FFT (EinFFT) for channel modeling by specific eigenvalue computations
  and uses the Mamba block for sequence modeling. Extensive performance studies
  across image and time-series benchmarks demonstrate that SiMBA outperforms
  existing SSMs, bridging the performance gap with state-of-the-art transformers.
  Notably, SiMBA establishes itself as the new state-of-the-art SSM on ImageNet
  and transfer learning benchmarks such as Stanford Car and Flower as well as
  task learning benchmarks as well as seven time series benchmark datasets. The
  project page is available on this website
  ~https://github.com/badripatro/Simba.
paper_ai_keywords: Transformers attention networks sequence mixing channel mixing MLPs State Space Models (SSMs) S4 Hippo Global Convolutions liquid S4 LRU Mega Mamba SiMBA Einstein FFT (EinFFT) channel modeling specific eigenvalue computations sequence modeling ImageNet Stanford Car Flower task learning benchmarks time series benchmark datasets