title: Jamba: A Hybrid Transformer-Mamba Language Model
summary: We present Jamba, a new base large language model based on a novel hybrid
  Transformer-Mamba mixture-of-experts (MoE) architecture. Specifically, Jamba
  interleaves blocks of Transformer and Mamba layers, enjoying the benefits of
  both model families. MoE is added in some of these layers to increase model
  capacity while keeping active parameter usage manageable. This flexible
  architecture allows resource- and objective-specific configurations. In the
  particular configuration we have implemented, we end up with a powerful model
  that fits in a single 80GB GPU. Built at large scale, Jamba provides high
  throughput and small memory footprint compared to vanilla Transformers, and at
  the same time state-of-the-art performance on standard language model
  benchmarks and long-context evaluations. Remarkably, the model presents strong
  results for up to 256K tokens context length. We study various architectural
  decisions, such as how to combine Transformer and Mamba layers, and how to mix
  experts, and show that some of them are crucial in large scale modeling. We
  also describe several interesting properties of these architectures which the
  training and evaluation of Jamba have revealed, and plan to release checkpoints
  from various ablation runs, to encourage further exploration of this novel
  architecture. We make the weights of our implementation of Jamba publicly
  available under a permissive license.
paper_authors._id: 660a2edb40e346fba5b51b82
paper_authors.name: Opher Lieber
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-25T07:34:41.626Z
paper_authors._id: 64e1bffd87a7332d25b79540
paper_authors.avatarUrl: /avatars/2205afcba52a00e51517a30448693c65.svg
paper_authors.fullname: Opher Lieber
paper_authors.type: user
paper_authors.user: opherlieber
paper_authors._id: 660a2edb40e346fba5b51b83
paper_authors.name: Barak Lenz
paper_authors._id: 660a2edb40e346fba5b51b84
paper_authors.name: Hofit Bata
paper_authors._id: 660a2edb40e346fba5b51b85
paper_authors.name: Gal Cohen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-02T11:04:25.248Z
paper_authors._id: 635aa711b13777d661f99a71
paper_authors.avatarUrl: /avatars/78409dd88639cb9e74c055223b7540d1.svg
paper_authors.fullname: Gal Cohen
paper_authors.type: user
paper_authors.user: galco
paper_authors._id: 660a2edb40e346fba5b51b86
paper_authors.name: Jhonathan Osin
paper_authors._id: 660a2edb40e346fba5b51b87
paper_authors.name: Itay Dalmedigos
paper_authors._id: 660a2edb40e346fba5b51b88
paper_authors.name: Erez Safahi
paper_authors._id: 660a2edb40e346fba5b51b89
paper_authors.name: Shaked Meirom
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-25T07:35:15.365Z
paper_authors._id: 65e7310d2519cf95214030fa
paper_authors.avatarUrl: /avatars/0c3124e8c8b40c4233244bac0ef605f1.svg
paper_authors.fullname: Shaked Meirom
paper_authors.type: user
paper_authors.user: ShakedM
paper_authors._id: 660a2edb40e346fba5b51b8a
paper_authors.name: Yonatan Belinkov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-25T07:35:22.149Z
paper_authors._id: 614c57f1ee44bcfe57b366d6
paper_authors.avatarUrl: /avatars/186a9aed84681246f48ed2a012c50def.svg
paper_authors.fullname: Yonatan Belinkov
paper_authors.type: user
paper_authors.user: belinkov
paper_authors._id: 660a2edb40e346fba5b51b8b
paper_authors.name: Shai Shalev-Shwartz
paper_authors._id: 660a2edb40e346fba5b51b8c
paper_authors.name: Omri Abend
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-25T07:35:35.699Z
paper_authors._id: 65375d628e37b02865e00265
paper_authors.avatarUrl: /avatars/ede8d84f60645512a6c22bbf3b6ade74.svg
paper_authors.fullname: Omri Abend
paper_authors.type: user
paper_authors.user: omriabnd
paper_authors._id: 660a2edb40e346fba5b51b8d
paper_authors.name: Raz Alon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-25T07:35:42.209Z
paper_authors._id: 65a7ca95e5ddfd7d1d97a6a0
paper_authors.avatarUrl: /avatars/84ebb95a07dd884e34f0170b07b1d652.svg
paper_authors.fullname: Raz Alon
paper_authors.type: user
paper_authors.user: RazAlon
paper_authors._id: 660a2edb40e346fba5b51b8e
paper_authors.name: Tomer Asida
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-02T11:04:16.537Z
paper_authors._id: 65f2d8dfab7d7db7f5d78900
paper_authors.avatarUrl: /avatars/feaa4973b6a6741c159bb2bc94d1ba59.svg
paper_authors.fullname: Tomer Asida
paper_authors.type: user
paper_authors.user: tomeras1
paper_authors._id: 660a2edb40e346fba5b51b8f
paper_authors.name: Amir Bergman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-25T07:35:49.115Z
paper_authors._id: 65fbfed32c813664be51d24b
paper_authors.avatarUrl: /avatars/acc6f4c5968003f7e2a587616fffd069.svg
paper_authors.fullname: Amir Bergman
paper_authors.type: user
paper_authors.user: amirbe
paper_authors._id: 660a2edb40e346fba5b51b90
paper_authors.name: Roman Glozman
paper_authors._id: 660a2edb40e346fba5b51b91
paper_authors.name: Michael Gokhman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-25T07:35:59.812Z
paper_authors._id: 60ede03f36e9ceda399a423e
paper_authors.avatarUrl: /avatars/da4794bfca8c430d271fcb50a19da28f.svg
paper_authors.fullname: Michael Gokhman
paper_authors.type: user
paper_authors.user: michael-go
paper_authors._id: 660a2edb40e346fba5b51b92
paper_authors.name: Avashalom Manevich
paper_authors._id: 660a2edb40e346fba5b51b93
paper_authors.name: Nir Ratner
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-25T07:36:08.655Z
paper_authors._id: 646630317ff8fcbef7d35079
paper_authors.avatarUrl: /avatars/a0650f7e9d8eb3cb01ba16fdf4616360.svg
paper_authors.fullname: Nir Ratner
paper_authors.type: user
paper_authors.user: Nirer
paper_authors._id: 660a2edb40e346fba5b51b94
paper_authors.name: Noam Rozen
paper_authors._id: 660a2edb40e346fba5b51b95
paper_authors.name: Erez Shwartz
paper_authors._id: 660a2edb40e346fba5b51b96
paper_authors.name: Mor Zusman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-25T07:36:21.962Z
paper_authors._id: 65e6d33bacf5bc4e4e18a1c1
paper_authors.avatarUrl: /avatars/b9325f34fd3275782b1ae487ec08d9a2.svg
paper_authors.fullname: Mor Zusman
paper_authors.type: user
paper_authors.user: mzusman
paper_authors._id: 660a2edb40e346fba5b51b97
paper_authors.name: Yoav Shoham
paper_title: Jamba: A Hybrid Transformer-Mamba Language Model
paper_summary: We present Jamba, a new base large language model based on a novel hybrid
  Transformer-Mamba mixture-of-experts (MoE) architecture. Specifically, Jamba
  interleaves blocks of Transformer and Mamba layers, enjoying the benefits of
  both model families. MoE is added in some of these layers to increase model
  capacity while keeping active parameter usage manageable. This flexible
  architecture allows resource- and objective-specific configurations. In the
  particular configuration we have implemented, we end up with a powerful model
  that fits in a single 80GB GPU. Built at large scale, Jamba provides high
  throughput and small memory footprint compared to vanilla Transformers, and at
  the same time state-of-the-art performance on standard language model
  benchmarks and long-context evaluations. Remarkably, the model presents strong
  results for up to 256K tokens context length. We study various architectural
  decisions, such as how to combine Transformer and Mamba layers, and how to mix
  experts, and show that some of them are crucial in large scale modeling. We
  also describe several interesting properties of these architectures which the
  training and evaluation of Jamba have revealed, and plan to release checkpoints
  from various ablation runs, to encourage further exploration of this novel
  architecture. We make the weights of our implementation of Jamba publicly
  available under a permissive license.
paper_ai_summary: Jamba, a hybrid Transformer-Mamba MoE model, achieves state-of-the-art performance with high throughput and minimal memory usage, supporting large context lengths.
paper_ai_keywords: Transformer Mamba mixture-of-experts (MoE) interleaving layers model capacity active parameter usage resource-specific configurations memory footprint long-context evaluations ablation runs