title: Falcon Mamba: The First Competitive Attention-free 7B Language Model
summary: In this technical report, we present Falcon Mamba 7B, a new base large
  language model based on the novel Mamba architecture. Falcon Mamba 7B is
  trained on 5.8 trillion tokens with carefully selected data mixtures. As a pure
  Mamba-based model, Falcon Mamba 7B surpasses leading open-weight models based
  on Transformers, such as Mistral 7B, Llama3.1 8B, and Falcon2 11B. It is on par
  with Gemma 7B and outperforms models with different architecture designs, such
  as RecurrentGemma 9B and RWKV-v6 Finch 7B/14B. Currently, Falcon Mamba 7B is
  the best-performing Mamba model in the literature at this scale, surpassing
  both existing Mamba and hybrid Mamba-Transformer models, according to the Open
  LLM Leaderboard. Due to its architecture, Falcon Mamba 7B is significantly
  faster at inference and requires substantially less memory for long sequence
  generation. Despite recent studies suggesting that hybrid Mamba-Transformer
  models outperform pure architecture designs, we demonstrate that even the pure
  Mamba design can achieve similar, or even superior results compared to the
  Transformer and hybrid designs. We make the weights of our implementation of
  Falcon Mamba 7B publicly available on
  https://huggingface.co/tiiuae/falcon-mamba-7b, under a permissive license.
paper_authors._id: 6707619fd24201cd8c5d381b
paper_authors.name: Jingwei Zuo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:00:20.786Z
paper_authors._id: 6460c3811db65f878513bcaf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6460c3811db65f878513bcaf/CRdJ8lXixDku3k8Rm5Stn.jpeg
paper_authors.fullname: Jingwei Zuo
paper_authors.type: user
paper_authors.user: JingweiZuo
paper_authors._id: 6707619fd24201cd8c5d381c
paper_authors.name: Maksim Velikanov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:00:15.197Z
paper_authors._id: 64670db15993aa7666cc6022
paper_authors.avatarUrl: /avatars/b68caad7e987c095b0cab4d9035aac25.svg
paper_authors.fullname: Maksim Velikanov
paper_authors.type: user
paper_authors.user: yellowvm
paper_authors._id: 6707619fd24201cd8c5d381d
paper_authors.name: Dhia Eddine Rhaiem
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-17T16:18:18.589Z
paper_authors._id: 664ddb00da286d1a60c298fc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/664ddb00da286d1a60c298fc/VchH_2fFjZxnBD_Ht10ED.jpeg
paper_authors.fullname: Rhaiem
paper_authors.type: user
paper_authors.user: DhiyaEddine
paper_authors._id: 6707619fd24201cd8c5d381e
paper_authors.name: Ilyas Chahed
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-23T11:12:07.400Z
paper_authors._id: 6697a9fb6d173ec7382e0392
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6697a9fb6d173ec7382e0392/Q_myrIBbdWI3RmEvEHrtQ.jpeg
paper_authors.fullname: Ilyas Chahed
paper_authors.type: user
paper_authors.user: IChahed
paper_authors._id: 6707619fd24201cd8c5d381f
paper_authors.name: Younes Belkada
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:00:24.878Z
paper_authors._id: 62441d1d9fdefb55a0b7d12c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1648631057413-noauth.png
paper_authors.fullname: Younes B
paper_authors.type: user
paper_authors.user: ybelkada
paper_authors._id: 6707619fd24201cd8c5d3820
paper_authors.name: Guillaume Kunsch
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T07:44:02.863Z
paper_authors._id: 62ebc3bb29bf16b52aee5f85
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62ebc3bb29bf16b52aee5f85/70akF1ZYIb_PCfESBTmP4.jpeg
paper_authors.fullname: Guillaume Kunsch
paper_authors.type: user
paper_authors.user: Gkunsch
paper_authors._id: 6707619fd24201cd8c5d3821
paper_authors.name: Hakim Hacid
paper_title: Falcon Mamba: The First Competitive Attention-free 7B Language Model
paper_summary: In this technical report, we present Falcon Mamba 7B, a new base large
  language model based on the novel Mamba architecture. Falcon Mamba 7B is
  trained on 5.8 trillion tokens with carefully selected data mixtures. As a pure
  Mamba-based model, Falcon Mamba 7B surpasses leading open-weight models based
  on Transformers, such as Mistral 7B, Llama3.1 8B, and Falcon2 11B. It is on par
  with Gemma 7B and outperforms models with different architecture designs, such
  as RecurrentGemma 9B and RWKV-v6 Finch 7B/14B. Currently, Falcon Mamba 7B is
  the best-performing Mamba model in the literature at this scale, surpassing
  both existing Mamba and hybrid Mamba-Transformer models, according to the Open
  LLM Leaderboard. Due to its architecture, Falcon Mamba 7B is significantly
  faster at inference and requires substantially less memory for long sequence
  generation. Despite recent studies suggesting that hybrid Mamba-Transformer
  models outperform pure architecture designs, we demonstrate that even the pure
  Mamba design can achieve similar, or even superior results compared to the
  Transformer and hybrid designs. We make the weights of our implementation of
  Falcon Mamba 7B publicly available on
  https://huggingface.co/tiiuae/falcon-mamba-7b, under a permissive license.
paper_ai_summary: Falcon Mamba 7B, a pure Mamba architecture language model, outperforms leading Transformer and hybrid models with faster inference and lower memory usage.
paper_ai_keywords: Mamba architecture large language model Transformers Mistral 7B Llama3.1 8B Falcon2 11B Gemma 7B RecurrentGemma 9B RWKV-v6 Finch 7B/14B Open LLM Leaderboard