Falcon Mamba: The First Competitive Attention-free 7B Language Model
In this technical report, we present Falcon Mamba 7B, a new base large
language model based on the novel Mamba architecture. Falcon Mamba 7B is
trained on 5.8 trillion tokens with carefully selected data mixtures. As a pure
Mamba-based model, Falcon Mamba 7B surpasses leading open-weight models based
on Transformers, such as Mistral 7B, Llama3.1 8B, and Falcon2 11B. It is on par
with Gemma 7B and outperforms models with different architecture designs, such
as RecurrentGemma 9B and RWKV-v6 Finch 7B/14B. Currently, Falcon Mamba 7B is
the best-performing Mamba model in the literature at this scale, surpassing
both existing Mamba and hybrid Mamba-Transformer models, according to the Open
LLM Leaderboard. Due to its architecture, Falcon Mamba 7B is significantly
faster at inference and requires substantially less memory for long sequence
generation. Despite recent studies suggesting that hybrid Mamba-Transformer
models outperform pure architecture designs, we demonstrate that even the pure
Mamba design can achieve similar, or even superior results compared to the
Transformer and hybrid designs. We make the weights of our implementation of
Falcon Mamba 7B publicly available on
https://huggingface.co/tiiuae/falcon-mamba-7b, under a permissive license.
6707619fd24201cd8c5d381b
Jingwei Zuo
claimed_verified
2024-10-10T08:00:20.786Z
6460c3811db65f878513bcaf
https://cdn-avatars.huggingface.co/v1/production/uploads/6460c3811db65f878513bcaf/CRdJ8lXixDku3k8Rm5Stn.jpeg
Jingwei Zuo
user
JingweiZuo
6707619fd24201cd8c5d381c
Maksim Velikanov
claimed_verified
2024-10-10T08:00:15.197Z
64670db15993aa7666cc6022
/avatars/b68caad7e987c095b0cab4d9035aac25.svg
Maksim Velikanov
user
yellowvm
6707619fd24201cd8c5d381d
Dhia Eddine Rhaiem
claimed_verified
2024-11-17T16:18:18.589Z
664ddb00da286d1a60c298fc
https://cdn-avatars.huggingface.co/v1/production/uploads/664ddb00da286d1a60c298fc/VchH_2fFjZxnBD_Ht10ED.jpeg
Rhaiem
user
DhiyaEddine
6707619fd24201cd8c5d381e
Ilyas Chahed
claimed_verified
2024-12-23T11:12:07.400Z
6697a9fb6d173ec7382e0392
https://cdn-avatars.huggingface.co/v1/production/uploads/6697a9fb6d173ec7382e0392/Q_myrIBbdWI3RmEvEHrtQ.jpeg
Ilyas Chahed
user
IChahed
6707619fd24201cd8c5d381f
Younes Belkada
claimed_verified
2024-10-10T08:00:24.878Z
62441d1d9fdefb55a0b7d12c
https://cdn-avatars.huggingface.co/v1/production/uploads/1648631057413-noauth.png
Younes B
user
ybelkada
6707619fd24201cd8c5d3820
Guillaume Kunsch
claimed_verified
2024-10-11T07:44:02.863Z
62ebc3bb29bf16b52aee5f85
https://cdn-avatars.huggingface.co/v1/production/uploads/62ebc3bb29bf16b52aee5f85/70akF1ZYIb_PCfESBTmP4.jpeg
Guillaume Kunsch
user
Gkunsch
6707619fd24201cd8c5d3821
Hakim Hacid
Falcon Mamba: The First Competitive Attention-free 7B Language Model
In this technical report, we present Falcon Mamba 7B, a new base large
language model based on the novel Mamba architecture. Falcon Mamba 7B is
trained on 5.8 trillion tokens with carefully selected data mixtures. As a pure
Mamba-based model, Falcon Mamba 7B surpasses leading open-weight models based
on Transformers, such as Mistral 7B, Llama3.1 8B, and Falcon2 11B. It is on par
with Gemma 7B and outperforms models with different architecture designs, such
as RecurrentGemma 9B and RWKV-v6 Finch 7B/14B. Currently, Falcon Mamba 7B is
the best-performing Mamba model in the literature at this scale, surpassing
both existing Mamba and hybrid Mamba-Transformer models, according to the Open
LLM Leaderboard. Due to its architecture, Falcon Mamba 7B is significantly
faster at inference and requires substantially less memory for long sequence
generation. Despite recent studies suggesting that hybrid Mamba-Transformer
models outperform pure architecture designs, we demonstrate that even the pure
Mamba design can achieve similar, or even superior results compared to the
Transformer and hybrid designs. We make the weights of our implementation of
Falcon Mamba 7B publicly available on
https://huggingface.co/tiiuae/falcon-mamba-7b, under a permissive license.
Falcon Mamba 7B, a pure Mamba architecture language model, outperforms leading Transformer and hybrid models with faster inference and lower memory usage.
Mamba architecture large language model Transformers Mistral 7B Llama3.1 8B Falcon2 11B Gemma 7B RecurrentGemma 9B RWKV-v6 Finch 7B/14B Open LLM Leaderboard