Jamba: Hybrid Transformer-Mamba Language Models
Barak Lenz Opher Lieber Alan Arazi Amir Bergman Avshalom Manevich Barak Peleg Ben Aviram Chen Almagor Clara Fridman Dan Padnos Daniel Gissin Daniel Jannai Dor Muhlgay Dor Zimberg Edden M. Gerber Elad Dolev Eran Krakovsky Erez Safahi Erez Schwartz Gal Cohen et al. (41 additional authors not shown)
We present Jamba, a novel hybrid Transformer-Mamba mixture-of-experts (MoE) architecture. Jamba interleaves blocks of Transformer and Mamba layers, enjoying the benefits of both model families. MoE is added in some of these layers to increase model capacity while keeping active parameter usage manageable. This flexible architecture allows resource- and objective-specific configurations. We implement two configurations: Jamba-1.5-Large, with 94B active parameters, and Jamba-1.5-mini, with 12B active parameters. Built at large scale, Jamba models provide high throughput and small memory footprint compared to vanilla Transformers, especially at long-context tasks, with an effective context length of 256K tokens, the largest amongst open-weight models. At the same time, they are also competitive on standard language modeling and chatbot benchmarks. We study various architectural decisions, such as how to combine Transformer and Mamba layers, and how to mix experts, and show that some of them are crucial in large scale modeling. To support cost-effective inference, we introduce ExpertsInt8, a novel quantization technique that allows fitting Jamba-1.5-Large on a machine with 8 80GB GPUs when processing 256K-token contexts without loss of quality. We also describe several interesting properties of this architecture that the training and evaluation of Jamba have revealed. The model weights are publicly available.
language models, state-space models, mamba, hybrid architecture, foundation models
Hybrid Transformer-Mamba models give better throughput and long-context capabilities at large scale