title: Zamba: A Compact 7B SSM Hybrid Model
summary: In this technical report, we present Zamba, a novel 7B SSM-transformer hybrid
  model which achieves competitive performance against leading open-weight models
  at a comparable scale. Zamba is trained on 1T tokens from openly available
  datasets and is the best non-transformer model at this scale. Zamba pioneers a
  unique architecture combining a Mamba backbone with a single shared attention
  module, thus obtaining the benefits of attention at minimal parameter cost. Due
  to its architecture, Zamba is significantly faster at inference than comparable
  transformer models and requires substantially less memory for generation of
  long sequences. Zamba is pretrained in two phases: the first phase is based on
  existing web datasets, while the second one consists of annealing the model
  over high-quality instruct and synthetic datasets, and is characterized by a
  rapid learning rate decay. We open-source the weights and all checkpoints for
  Zamba, through both phase 1 and annealing phases.
paper_authors._id: 66555178c3b51bcf54afc562
paper_authors.name: Paolo Glorioso
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T09:40:40.762Z
paper_authors._id: 64e40335c0edca443ef8af3e
paper_authors.avatarUrl: /avatars/c42a72a448dcb9e8bb3a0b41d1b6397a.svg
paper_authors.fullname: Paolo Glorioso
paper_authors.type: user
paper_authors.user: pglo
paper_authors._id: 66555178c3b51bcf54afc563
paper_authors.name: Quentin Anthony
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T09:47:35.906Z
paper_authors._id: 65bc13717c6ad1994b6619e9
paper_authors.avatarUrl: /avatars/8e97e25e2de30f6bea9f9bc94418ff0f.svg
paper_authors.fullname: Quentin Anthony
paper_authors.type: user
paper_authors.user: qanthony-z
paper_authors._id: 66555178c3b51bcf54afc564
paper_authors.name: Yury Tokpanov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T09:47:43.700Z
paper_authors._id: 65455aca468722e935103b17
paper_authors.avatarUrl: /avatars/29786a7df6541aa1e5d408eb06b0bd96.svg
paper_authors.fullname: Yury Tokpanov
paper_authors.type: user
paper_authors.user: yury-zyphra
paper_authors._id: 66555178c3b51bcf54afc565
paper_authors.name: James Whittington
paper_authors._id: 66555178c3b51bcf54afc566
paper_authors.name: Jonathan Pilault
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T09:48:00.516Z
paper_authors._id: 65f1a7b2ea222a82562de3e9
paper_authors.avatarUrl: /avatars/7f62489688acf30d4edc7f9d2036b14b.svg
paper_authors.fullname: Jonathan Pilault
paper_authors.type: user
paper_authors.user: jpilault
paper_authors._id: 66555178c3b51bcf54afc567
paper_authors.name: Adam Ibrahim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T09:48:06.540Z
paper_authors._id: 648743f05aed4a7b0bf2e3a6
paper_authors.avatarUrl: /avatars/e6f3b35085b1b8cb2c231516d1fea70a.svg
paper_authors.fullname: Adam Ibrahim
paper_authors.type: user
paper_authors.user: adami1
paper_authors._id: 66555178c3b51bcf54afc568
paper_authors.name: Beren Millidge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T09:48:12.249Z
paper_authors._id: 65c05e75c084467acab2f84a
paper_authors.avatarUrl: /avatars/255e98fe436d9d9252975e842755c0ff.svg
paper_authors.fullname: Beren Millidge
paper_authors.type: user
paper_authors.user: BerenMillidge
paper_title: Zamba: A Compact 7B SSM Hybrid Model
paper_summary: In this technical report, we present Zamba, a novel 7B SSM-transformer hybrid
  model which achieves competitive performance against leading open-weight models
  at a comparable scale. Zamba is trained on 1T tokens from openly available
  datasets and is the best non-transformer model at this scale. Zamba pioneers a
  unique architecture combining a Mamba backbone with a single shared attention
  module, thus obtaining the benefits of attention at minimal parameter cost. Due
  to its architecture, Zamba is significantly faster at inference than comparable
  transformer models and requires substantially less memory for generation of
  long sequences. Zamba is pretrained in two phases: the first phase is based on
  existing web datasets, while the second one consists of annealing the model
  over high-quality instruct and synthetic datasets, and is characterized by a
  rapid learning rate decay. We open-source the weights and all checkpoints for
  Zamba, through both phase 1 and annealing phases.
paper_ai_summary: Zamba, a 7B SSM-transformer hybrid model, achieves competitive performance with minimal parameter cost and efficient inference by combining a Mamba backbone with a shared attention module.
paper_ai_keywords: SSM-transformer Mamba backbone shared attention module inference speed memory efficiency pretraining annealing instruct datasets synthetic datasets rapid learning rate decay