Jamba: A Hybrid Transformer-Mamba Language Model
We present Jamba, a new base large language model based on a novel hybrid
Transformer-Mamba mixture-of-experts (MoE) architecture. Specifically, Jamba
interleaves blocks of Transformer and Mamba layers, enjoying the benefits of
both model families. MoE is added in some of these layers to increase model
capacity while keeping active parameter usage manageable. This flexible
architecture allows resource- and objective-specific configurations. In the
particular configuration we have implemented, we end up with a powerful model
that fits in a single 80GB GPU. Built at large scale, Jamba provides high
throughput and small memory footprint compared to vanilla Transformers, and at
the same time state-of-the-art performance on standard language model
benchmarks and long-context evaluations. Remarkably, the model presents strong
results for up to 256K tokens context length. We study various architectural
decisions, such as how to combine Transformer and Mamba layers, and how to mix
experts, and show that some of them are crucial in large scale modeling. We
also describe several interesting properties of these architectures which the
training and evaluation of Jamba have revealed, and plan to release checkpoints
from various ablation runs, to encourage further exploration of this novel
architecture. We make the weights of our implementation of Jamba publicly
available under a permissive license.
660a2edb40e346fba5b51b82
Opher Lieber
admin_assigned
2024-04-25T07:34:41.626Z
64e1bffd87a7332d25b79540
/avatars/2205afcba52a00e51517a30448693c65.svg
Opher Lieber
user
opherlieber
660a2edb40e346fba5b51b83
Barak Lenz
660a2edb40e346fba5b51b84
Hofit Bata
660a2edb40e346fba5b51b85
Gal Cohen
claimed_verified
2024-04-02T11:04:25.248Z
635aa711b13777d661f99a71
/avatars/78409dd88639cb9e74c055223b7540d1.svg
Gal Cohen
user
galco
660a2edb40e346fba5b51b86
Jhonathan Osin
660a2edb40e346fba5b51b87
Itay Dalmedigos
660a2edb40e346fba5b51b88
Erez Safahi
660a2edb40e346fba5b51b89
Shaked Meirom
admin_assigned
2024-04-25T07:35:15.365Z
65e7310d2519cf95214030fa
/avatars/0c3124e8c8b40c4233244bac0ef605f1.svg
Shaked Meirom
user
ShakedM
660a2edb40e346fba5b51b8a
Yonatan Belinkov
admin_assigned
2024-04-25T07:35:22.149Z
614c57f1ee44bcfe57b366d6
/avatars/186a9aed84681246f48ed2a012c50def.svg
Yonatan Belinkov
user
belinkov
660a2edb40e346fba5b51b8b
Shai Shalev-Shwartz
660a2edb40e346fba5b51b8c
Omri Abend
admin_assigned
2024-04-25T07:35:35.699Z
65375d628e37b02865e00265
/avatars/ede8d84f60645512a6c22bbf3b6ade74.svg
Omri Abend
user
omriabnd
660a2edb40e346fba5b51b8d
Raz Alon
admin_assigned
2024-04-25T07:35:42.209Z
65a7ca95e5ddfd7d1d97a6a0
/avatars/84ebb95a07dd884e34f0170b07b1d652.svg
Raz Alon
user
RazAlon
660a2edb40e346fba5b51b8e
Tomer Asida
claimed_verified
2024-04-02T11:04:16.537Z
65f2d8dfab7d7db7f5d78900
/avatars/feaa4973b6a6741c159bb2bc94d1ba59.svg
Tomer Asida
user
tomeras1
660a2edb40e346fba5b51b8f
Amir Bergman
admin_assigned
2024-04-25T07:35:49.115Z
65fbfed32c813664be51d24b
/avatars/acc6f4c5968003f7e2a587616fffd069.svg
Amir Bergman
user
amirbe
660a2edb40e346fba5b51b90
Roman Glozman
660a2edb40e346fba5b51b91
Michael Gokhman
admin_assigned
2024-04-25T07:35:59.812Z
60ede03f36e9ceda399a423e
/avatars/da4794bfca8c430d271fcb50a19da28f.svg
Michael Gokhman
user
michael-go
660a2edb40e346fba5b51b92
Avashalom Manevich
660a2edb40e346fba5b51b93
Nir Ratner
admin_assigned
2024-04-25T07:36:08.655Z
646630317ff8fcbef7d35079
/avatars/a0650f7e9d8eb3cb01ba16fdf4616360.svg
Nir Ratner
user
Nirer
660a2edb40e346fba5b51b94
Noam Rozen
660a2edb40e346fba5b51b95
Erez Shwartz
660a2edb40e346fba5b51b96
Mor Zusman
admin_assigned
2024-04-25T07:36:21.962Z
65e6d33bacf5bc4e4e18a1c1
/avatars/b9325f34fd3275782b1ae487ec08d9a2.svg
Mor Zusman
user
mzusman
660a2edb40e346fba5b51b97
Yoav Shoham
Jamba: A Hybrid Transformer-Mamba Language Model
We present Jamba, a new base large language model based on a novel hybrid
Transformer-Mamba mixture-of-experts (MoE) architecture. Specifically, Jamba
interleaves blocks of Transformer and Mamba layers, enjoying the benefits of
both model families. MoE is added in some of these layers to increase model
capacity while keeping active parameter usage manageable. This flexible
architecture allows resource- and objective-specific configurations. In the
particular configuration we have implemented, we end up with a powerful model
that fits in a single 80GB GPU. Built at large scale, Jamba provides high
throughput and small memory footprint compared to vanilla Transformers, and at
the same time state-of-the-art performance on standard language model
benchmarks and long-context evaluations. Remarkably, the model presents strong
results for up to 256K tokens context length. We study various architectural
decisions, such as how to combine Transformer and Mamba layers, and how to mix
experts, and show that some of them are crucial in large scale modeling. We
also describe several interesting properties of these architectures which the
training and evaluation of Jamba have revealed, and plan to release checkpoints
from various ablation runs, to encourage further exploration of this novel
architecture. We make the weights of our implementation of Jamba publicly
available under a permissive license.
Jamba, a hybrid Transformer-Mamba MoE model, achieves state-of-the-art performance with high throughput and minimal memory usage, supporting large context lengths.
Transformer Mamba mixture-of-experts (MoE) interleaving layers model capacity active parameter usage resource-specific configurations memory footprint long-context evaluations ablation runs