title: Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning
    Tasks
summary: State-space models (SSMs), such as Mamba Gu & Dao (2034), have been proposed
  as alternatives to Transformer networks in language modeling, by incorporating
  gating, convolutions, and input-dependent token selection to mitigate the
  quadratic cost of multi-head attention. Although SSMs exhibit competitive
  performance, their in-context learning (ICL) capabilities, a remarkable
  emergent property of modern language models that enables task execution without
  parameter optimization, remain underexplored compared to Transformers. In this
  study, we evaluate the ICL performance of SSMs, focusing on Mamba, against
  Transformer models across various tasks. Our results show that SSMs perform
  comparably to Transformers in standard regression ICL tasks, while
  outperforming them in tasks like sparse parity learning. However, SSMs fall
  short in tasks involving non-standard retrieval functionality. To address these
  limitations, we introduce a hybrid model, \variant, that combines Mamba with
  attention blocks, surpassing individual models in tasks where they struggle
  independently. Our findings suggest that hybrid architectures offer promising
  avenues for enhancing ICL in language models.
paper_authors._id: 65c2f90a82e4546db95c26d8
paper_authors.name: Jongho Park
paper_authors._id: 65c2f90a82e4546db95c26d9
paper_authors.name: Jaeseung Park
paper_authors._id: 65c2f90a82e4546db95c26da
paper_authors.name: Zheyang Xiong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T07:46:44.350Z
paper_authors._id: 67084dbb652deb3c09d8f633
paper_authors.avatarUrl: /avatars/6489cc414ec6b7be3523e335ba14888f.svg
paper_authors.fullname: Zheyang Xiong
paper_authors.type: user
paper_authors.user: zheyang-hf
paper_authors._id: 65c2f90a82e4546db95c26db
paper_authors.name: Nayoung Lee
paper_authors._id: 65c2f90a82e4546db95c26dc
paper_authors.name: Jaewoong Cho
paper_authors._id: 65c2f90a82e4546db95c26dd
paper_authors.name: Samet Oymak
paper_authors._id: 65c2f90a82e4546db95c26de
paper_authors.name: Kangwook Lee
paper_authors._id: 65c2f90a82e4546db95c26df
paper_authors.name: Dimitris Papailiopoulos
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-02-07T03:29:15.708Z
paper_authors._id: 64428cf07feb866811b0be6a
paper_authors.avatarUrl: /avatars/f30d339a02ce070c7e77315b59e20359.svg
paper_authors.fullname: Dimitris Papailiopoulos
paper_authors.type: user
paper_authors.user: anadim
paper_title: Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning
    Tasks
paper_summary: State-space models (SSMs), such as Mamba Gu & Dao (2034), have been proposed
  as alternatives to Transformer networks in language modeling, by incorporating
  gating, convolutions, and input-dependent token selection to mitigate the
  quadratic cost of multi-head attention. Although SSMs exhibit competitive
  performance, their in-context learning (ICL) capabilities, a remarkable
  emergent property of modern language models that enables task execution without
  parameter optimization, remain underexplored compared to Transformers. In this
  study, we evaluate the ICL performance of SSMs, focusing on Mamba, against
  Transformer models across various tasks. Our results show that SSMs perform
  comparably to Transformers in standard regression ICL tasks, while
  outperforming them in tasks like sparse parity learning. However, SSMs fall
  short in tasks involving non-standard retrieval functionality. To address these
  limitations, we introduce a hybrid model, \variant, that combines Mamba with
  attention blocks, surpassing individual models in tasks where they struggle
  independently. Our findings suggest that hybrid architectures offer promising
  avenues for enhancing ICL in language models.
paper_ai_keywords: state-space models Mamba Gu & Dao gating convolutions input-dependent token selection multi-head attention in-context learning task execution parameter optimization standard regression Tasks sparse parity learning non-standard retrieval functionality attention blocks hybrid architectures