title: Samba: Simple Hybrid State Space Models for Efficient Unlimited Context
    Language Modeling
summary: Efficiently modeling sequences with infinite context length has been a
  long-standing problem. Past works suffer from either the quadratic computation
  complexity or the limited extrapolation ability on length generalization. In
  this work, we present Samba, a simple hybrid architecture that layer-wise
  combines Mamba, a selective State Space Model (SSM), with Sliding Window
  Attention (SWA). Samba selectively compresses a given sequence into recurrent
  hidden states while still maintaining the ability to precisely recall memories
  with the attention mechanism. We scale Samba up to 3.8B parameters with 3.2T
  training tokens and show that Samba substantially outperforms the
  state-of-the-art models based on pure attention or SSMs on a wide range of
  benchmarks. When trained on 4K length sequences, Samba can be efficiently
  extrapolated to 256K context length with perfect memory recall and show
  improved token predictions up to 1M context length. As a linear-time sequence
  model, Samba enjoys a 3.73x higher throughput compared to Transformers with
  grouped-query attention when processing user prompts of 128K length, and 3.64x
  speedup when generating 64K tokens with unlimited streaming. A sample
  implementation of Samba is publicly available in
  https://github.com/microsoft/Samba.
paper_authors._id: 66690da983408bb0da3535da
paper_authors.name: Liliang Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-12T07:29:37.990Z
paper_authors._id: 63815eff4761ddfa00903762
paper_authors.avatarUrl: /avatars/3419b239d42e091586f1c51b526d88e5.svg
paper_authors.fullname: Liliang Ren
paper_authors.type: user
paper_authors.user: renll
paper_authors._id: 66690da983408bb0da3535db
paper_authors.name: Yang Liu
paper_authors._id: 66690da983408bb0da3535dc
paper_authors.name: Yadong Lu
paper_authors._id: 66690da983408bb0da3535dd
paper_authors.name: Yelong Shen
paper_authors._id: 66690da983408bb0da3535de
paper_authors.name: Chen Liang
paper_authors._id: 66690da983408bb0da3535df
paper_authors.name: Weizhu Chen
paper_title: Samba: Simple Hybrid State Space Models for Efficient Unlimited Context
    Language Modeling
paper_summary: Efficiently modeling sequences with infinite context length has been a
  long-standing problem. Past works suffer from either the quadratic computation
  complexity or the limited extrapolation ability on length generalization. In
  this work, we present Samba, a simple hybrid architecture that layer-wise
  combines Mamba, a selective State Space Model (SSM), with Sliding Window
  Attention (SWA). Samba selectively compresses a given sequence into recurrent
  hidden states while still maintaining the ability to precisely recall memories
  with the attention mechanism. We scale Samba up to 3.8B parameters with 3.2T
  training tokens and show that Samba substantially outperforms the
  state-of-the-art models based on pure attention or SSMs on a wide range of
  benchmarks. When trained on 4K length sequences, Samba can be efficiently
  extrapolated to 256K context length with perfect memory recall and show
  improved token predictions up to 1M context length. As a linear-time sequence
  model, Samba enjoys a 3.73x higher throughput compared to Transformers with
  grouped-query attention when processing user prompts of 128K length, and 3.64x
  speedup when generating 64K tokens with unlimited streaming. A sample
  implementation of Samba is publicly available in
  https://github.com/microsoft/Samba.
paper_ai_keywords: State Space Model (SSM) Sliding Window Attention (SWA) recurrent hidden states attention mechanism Samba perfect memory recall token predictions linear-time sequence model Transformers grouped-query attention throughput speedup unlimited streaming