Jamba-1.5: Hybrid Transformer-Mamba Models at Scale
We present Jamba-1.5, new instruction-tuned large language models based on
our Jamba architecture. Jamba is a hybrid Transformer-Mamba mixture of experts
architecture, providing high throughput and low memory usage across context
lengths, while retaining the same or better quality as Transformer models. We
release two model sizes: Jamba-1.5-Large, with 94B active parameters, and
Jamba-1.5-Mini, with 12B active parameters. Both models are fine-tuned for a
variety of conversational and instruction-following capabilties, and have an
effective context length of 256K tokens, the largest amongst open-weight
models. To support cost-effective inference, we introduce ExpertsInt8, a novel
quantization technique that allows fitting Jamba-1.5-Large on a machine with 8
80GB GPUs when processing 256K-token contexts without loss of quality. When
evaluated on a battery of academic and chatbot benchmarks, Jamba-1.5 models
achieve excellent results while providing high throughput and outperforming
other open-weight models on long-context benchmarks. The model weights for both
sizes are publicly available under the Jamba Open Model License and we release
ExpertsInt8 as open source.
66c801c3d5a538bbb9e6f962
Jamba Team
66c801c3d5a538bbb9e6f963
Barak Lenz
66c801c3d5a538bbb9e6f964
Alan Arazi
claimed_verified
2025-05-26T14:09:52.602Z
60d84af7eac5e05d4594f010
https://cdn-avatars.huggingface.co/v1/production/uploads/60d84af7eac5e05d4594f010/KnGxUR7OUOAGg0S67tRaY.png
Alan Arazi
user
alana89
66c801c3d5a538bbb9e6f965
Amir Bergman
66c801c3d5a538bbb9e6f966
Avshalom Manevich
claimed_verified
2024-10-07T07:30:30.260Z
6311d40c41ebdfa5f1196997
https://cdn-avatars.huggingface.co/v1/production/uploads/6311d40c41ebdfa5f1196997/1TkHAAjtsO6oyBipyRi4n.png
Avshalom Manevich
user
avshalomman
66c801c3d5a538bbb9e6f967
Barak Peleg
66c801c3d5a538bbb9e6f968
Ben Aviram
66c801c3d5a538bbb9e6f969
Chen Almagor
66c801c3d5a538bbb9e6f96a
Clara Fridman
66c801c3d5a538bbb9e6f96b
Dan Padnos
66c801c3d5a538bbb9e6f96c
Daniel Gissin
66c801c3d5a538bbb9e6f96d
Daniel Jannai
66c801c3d5a538bbb9e6f96e
Dor Muhlgay
66c801c3d5a538bbb9e6f96f
Dor Zimberg
66c801c3d5a538bbb9e6f970
Edden M Gerber
66c801c3d5a538bbb9e6f971
Elad Dolev
66c801c3d5a538bbb9e6f972
Eran Krakovsky
66c801c3d5a538bbb9e6f973
Erez Safahi
66c801c3d5a538bbb9e6f974
Erez Schwartz
66c801c3d5a538bbb9e6f975
Gal Cohen
66c801c3d5a538bbb9e6f976
Gal Shachaf
66c801c3d5a538bbb9e6f977
Haim Rozenblum
66c801c3d5a538bbb9e6f978
Hofit Bata
66c801c3d5a538bbb9e6f979
Ido Blass
66c801c3d5a538bbb9e6f97a
Inbal Magar
66c801c3d5a538bbb9e6f97b
Itay Dalmedigos
66c801c3d5a538bbb9e6f97c
Jhonathan Osin
66c801c3d5a538bbb9e6f97d
Julie Fadlon
66c801c3d5a538bbb9e6f97e
Maria Rozman
66c801c3d5a538bbb9e6f97f
Matan Danos
66c801c3d5a538bbb9e6f980
Michael Gokhman
claimed_verified
2024-08-27T07:44:04.708Z
60ede03f36e9ceda399a423e
/avatars/da4794bfca8c430d271fcb50a19da28f.svg
Michael Gokhman
user
michael-go
66c801c3d5a538bbb9e6f981
Mor Zusman
66c801c3d5a538bbb9e6f982
Naama Gidron
66c801c3d5a538bbb9e6f983
Nir Ratner
66c801c3d5a538bbb9e6f984
Noam Gat
66c801c3d5a538bbb9e6f985
Noam Rozen
66c801c3d5a538bbb9e6f986
Oded Fried
66c801c3d5a538bbb9e6f987
Ohad Leshno
66c801c3d5a538bbb9e6f988
Omer Antverg
66c801c3d5a538bbb9e6f989
Omri Abend
66c801c3d5a538bbb9e6f98a
Opher Lieber
66c801c3d5a538bbb9e6f98b
Or Dagan
66c801c3d5a538bbb9e6f98c
Orit Cohavi
66c801c3d5a538bbb9e6f98d
Raz Alon
66c801c3d5a538bbb9e6f98e
Ro'i Belson
66c801c3d5a538bbb9e6f98f
Roi Cohen
claimed_verified
2024-09-02T08:04:08.536Z
64b7b79a8ba7d6c922d75144
https://cdn-avatars.huggingface.co/v1/production/uploads/64b7b79a8ba7d6c922d75144/S6WUOCj1_bKc4Gv62uSaa.jpeg
Roi Cohen
user
roicohennn
66c801c3d5a538bbb9e6f990
Rom Gilad
66c801c3d5a538bbb9e6f991
Roman Glozman
66c801c3d5a538bbb9e6f992
Shahar Lev
66c801c3d5a538bbb9e6f993
Shaked Meirom
66c801c3d5a538bbb9e6f994
Tal Delbari
66c801c3d5a538bbb9e6f995
Tal Ness
66c801c3d5a538bbb9e6f996
Tomer Asida
claimed_verified
2024-08-27T07:43:57.165Z
65f2d8dfab7d7db7f5d78900
/avatars/feaa4973b6a6741c159bb2bc94d1ba59.svg
Tomer Asida
user
tomeras1
66c801c3d5a538bbb9e6f997
Tom Ben Gal
claimed_verified
2024-08-29T15:22:12.956Z
65e6e40b5abdebc53059f2bf
/avatars/cd71b74d4d237f869cdbbff3361e88dd.svg
Tom Ben Gal
user
tombg
66c801c3d5a538bbb9e6f998
Tom Braude
66c801c3d5a538bbb9e6f999
Uriya Pumerantz
66c801c3d5a538bbb9e6f99a
Yehoshua Cohen
66c801c3d5a538bbb9e6f99b
Yonatan Belinkov
66c801c3d5a538bbb9e6f99c
Yuval Globerson
66c801c3d5a538bbb9e6f99d
Yuval Peleg Levy
66c801c3d5a538bbb9e6f99e
Yoav Shoham
Jamba-1.5: Hybrid Transformer-Mamba Models at Scale
We present Jamba-1.5, new instruction-tuned large language models based on
our Jamba architecture. Jamba is a hybrid Transformer-Mamba mixture of experts
architecture, providing high throughput and low memory usage across context
lengths, while retaining the same or better quality as Transformer models. We
release two model sizes: Jamba-1.5-Large, with 94B active parameters, and
Jamba-1.5-Mini, with 12B active parameters. Both models are fine-tuned for a
variety of conversational and instruction-following capabilties, and have an
effective context length of 256K tokens, the largest amongst open-weight
models. To support cost-effective inference, we introduce ExpertsInt8, a novel
quantization technique that allows fitting Jamba-1.5-Large on a machine with 8
80GB GPUs when processing 256K-token contexts without loss of quality. When
evaluated on a battery of academic and chatbot benchmarks, Jamba-1.5 models
achieve excellent results while providing high throughput and outperforming
other open-weight models on long-context benchmarks. The model weights for both
sizes are publicly available under the Jamba Open Model License and we release
ExpertsInt8 as open source.
Jamba-1.5, a hybrid Transformer-Mamba model, offers high throughput and low memory usage while achieving high quality in instruction-following tasks, supported by ExpertsInt8 quantization.
Transformer Mamba mixture of experts Jamba architecture ExpertsInt8 quantization technique