title: Mixtral of Experts
summary: We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model.
  Mixtral has the same architecture as Mistral 7B, with the difference that each
  layer is composed of 8 feedforward blocks (i.e. experts). For every token, at
  each layer, a router network selects two experts to process the current state
  and combine their outputs. Even though each token only sees two experts, the
  selected experts can be different at each timestep. As a result, each token has
  access to 47B parameters, but only uses 13B active parameters during inference.
  Mixtral was trained with a context size of 32k tokens and it outperforms or
  matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular,
  Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and
  multilingual benchmarks. We also provide a model fine-tuned to follow
  instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo,
  Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both
  the base and instruct models are released under the Apache 2.0 license.
paper_authors._id: 659cb2f4e942a8f717b442ac
paper_authors.name: Albert Q. Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T07:56:34.904Z
paper_authors._id: 6137dff56ecbbda5c86755e9
paper_authors.avatarUrl: /avatars/1aadefd3acb7ad24df15463dadceb7d4.svg
paper_authors.fullname: Albert Q Jiang
paper_authors.type: user
paper_authors.user: aqj213
paper_authors._id: 659cb2f4e942a8f717b442ad
paper_authors.name: Alexandre Sablayrolles
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T07:58:15.229Z
paper_authors._id: 6391c7456176fbc67b9ecd2c
paper_authors.avatarUrl: /avatars/4aa9be94d8284118b8018362abf11f01.svg
paper_authors.fullname: Alexandre Sablayrolles
paper_authors.type: user
paper_authors.user: alexsablay
paper_authors._id: 659cb2f4e942a8f717b442ae
paper_authors.name: Antoine Roux
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T07:58:22.438Z
paper_authors._id: 61643ec44175c9b7c811358a
paper_authors.avatarUrl: /avatars/ef445c8c2605ce8ed48b0cd228a2b0b7.svg
paper_authors.fullname: Antoine Roux
paper_authors.type: user
paper_authors.user: antwanroux
paper_authors._id: 659cb2f4e942a8f717b442af
paper_authors.name: Arthur Mensch
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T07:58:28.516Z
paper_authors._id: 645d28525ebf379fd6d9ae41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/645d28525ebf379fd6d9ae41/qVEKQk4gPlumsZE3ZorWG.jpeg
paper_authors.fullname: Arthur Mensch
paper_authors.type: user
paper_authors.user: arthurmensch
paper_authors._id: 659cb2f4e942a8f717b442b0
paper_authors.name: Blanche Savary
paper_authors._id: 659cb2f4e942a8f717b442b1
paper_authors.name: Chris Bamford
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T07:58:42.734Z
paper_authors._id: 6304b9a2bad6ce7fc026cab6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661254040723-noauth.png
paper_authors.fullname: Chris Bamford
paper_authors.type: user
paper_authors.user: Bam4d
paper_authors._id: 659cb2f4e942a8f717b442b2
paper_authors.name: Devendra Singh Chaplot
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T07:58:49.031Z
paper_authors._id: 65143e1c4f08b815c8db57a0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65143e1c4f08b815c8db57a0/JqkwKiJmLFRkH0NK3L8XH.jpeg
paper_authors.fullname: Devendra Singh Chaplot
paper_authors.type: user
paper_authors.user: devendrachaplot
paper_authors._id: 659cb2f4e942a8f717b442b3
paper_authors.name: Diego de las Casas
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T07:58:55.362Z
paper_authors._id: 657608c427c0e2fd6c894ced
paper_authors.avatarUrl: /avatars/7f1ca5f75ca03cec69560584496558c9.svg
paper_authors.fullname: Diego de las Casas
paper_authors.type: user
paper_authors.user: diegolascasas
paper_authors._id: 659cb2f4e942a8f717b442b4
paper_authors.name: Emma Bou Hanna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T07:59:37.471Z
paper_authors._id: 655dc9138e56f09d55197cac
paper_authors.avatarUrl: /avatars/764edb5c5e1003eb71cf6f1bea563f4b.svg
paper_authors.fullname: Emma Bou Hanna
paper_authors.type: user
paper_authors.user: EmmaBH
paper_authors._id: 659cb2f4e942a8f717b442b5
paper_authors.name: Florian Bressand
paper_authors._id: 659cb2f4e942a8f717b442b6
paper_authors.name: Gianna Lengyel
paper_authors._id: 659cb2f4e942a8f717b442b7
paper_authors.name: Guillaume Bour
paper_authors._id: 659cb2f4e942a8f717b442b8
paper_authors.name: Guillaume Lample
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T08:00:40.393Z
paper_authors._id: 64882fc9508d0656d43dd5b3
paper_authors.avatarUrl: /avatars/c3bc3656e4ff05663b54c6dd4f241891.svg
paper_authors.fullname: Guillaume Lample
paper_authors.type: user
paper_authors.user: glample
paper_authors._id: 659cb2f4e942a8f717b442b9
paper_authors.name: Lélio Renard Lavaud
paper_authors._id: 659cb2f4e942a8f717b442ba
paper_authors.name: Lucile Saulnier
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T08:01:14.405Z
paper_authors._id: 650cb0ebc705442b280160c4
paper_authors.avatarUrl: /avatars/c5abfcddfa66505085178fa430779ed0.svg
paper_authors.fullname: Lucile Saulnier
paper_authors.type: user
paper_authors.user: LucileSaulnier
paper_authors._id: 659cb2f4e942a8f717b442bb
paper_authors.name: Marie-Anne Lachaux
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T08:01:26.578Z
paper_authors._id: 651e7ab594de702b4aa8ae51
paper_authors.avatarUrl: /avatars/f097d3ca30e62567e600e85ca50e9a9d.svg
paper_authors.fullname: marie-anne lachaux
paper_authors.type: user
paper_authors.user: malachaux
paper_authors._id: 659cb2f4e942a8f717b442bc
paper_authors.name: Pierre Stock
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T08:01:33.210Z
paper_authors._id: 650c993d2751c84306aba92b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/650c993d2751c84306aba92b/uS04hveXAu6CEszCy-8_G.jpeg
paper_authors.fullname: Pierre Stock
paper_authors.type: user
paper_authors.user: pstock
paper_authors._id: 659cb2f4e942a8f717b442bd
paper_authors.name: Sandeep Subramanian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T08:01:40.422Z
paper_authors._id: 627bf27cf19c5eb46d54cea8
paper_authors.avatarUrl: /avatars/b8ca0b4e841858c1d234671187234f56.svg
paper_authors.fullname: Sandeep Subramanian
paper_authors.type: user
paper_authors.user: MaximumEntropy
paper_authors._id: 659cb2f4e942a8f717b442be
paper_authors.name: Sophia Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T08:03:00.989Z
paper_authors._id: 6320c05a145cfa4c04cb4359
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6320c05a145cfa4c04cb4359/jLYLrlc_LZQMi3yCrlfCi.jpeg
paper_authors.fullname: Sophia Yang
paper_authors.type: user
paper_authors.user: sophiamyang
paper_authors._id: 659cb2f4e942a8f717b442bf
paper_authors.name: Szymon Antoniak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T08:03:13.031Z
paper_authors._id: 63f767c4bd28622c9b984637
paper_authors.avatarUrl: /avatars/126a23ca5b3adc89418b7a53b21e50e6.svg
paper_authors.fullname: Szymon Antoniak
paper_authors.type: user
paper_authors.user: Simontwice
paper_authors._id: 659cb2f4e942a8f717b442c0
paper_authors.name: Teven Le Scao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-09T08:03:20.358Z
paper_authors._id: 5e67bed6100906368940747b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1583858339980-5e67bed6100906368940747b.jpeg
paper_authors.fullname: Teven Le Scao
paper_authors.type: user
paper_authors.user: teven
paper_authors._id: 659cb2f4e942a8f717b442c1
paper_authors.name: Théophile Gervet
paper_authors._id: 659cb2f4e942a8f717b442c2
paper_authors.name: Thibaut Lavril
paper_authors._id: 659cb2f4e942a8f717b442c3
paper_authors.name: Thomas Wang
paper_authors._id: 659cb2f4e942a8f717b442c4
paper_authors.name: Timothée Lacroix
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-15T09:53:10.733Z
paper_authors._id: 6503364ceda77aafd4a68a48
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6503364ceda77aafd4a68a48/POQqon4zWQEpFTPzHS4uu.png
paper_authors.fullname: timothee lacroix
paper_authors.type: user
paper_authors.user: timlacroix
paper_authors._id: 659cb2f4e942a8f717b442c5
paper_authors.name: William El Sayed
paper_title: Mixtral of Experts
paper_summary: We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model.
  Mixtral has the same architecture as Mistral 7B, with the difference that each
  layer is composed of 8 feedforward blocks (i.e. experts). For every token, at
  each layer, a router network selects two experts to process the current state
  and combine their outputs. Even though each token only sees two experts, the
  selected experts can be different at each timestep. As a result, each token has
  access to 47B parameters, but only uses 13B active parameters during inference.
  Mixtral was trained with a context size of 32k tokens and it outperforms or
  matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular,
  Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and
  multilingual benchmarks. We also provide a model fine-tuned to follow
  instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo,
  Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both
  the base and instruct models are released under the Apache 2.0 license.
paper_ai_summary: Mixtral 8x7B, a Sparse Mixture of Experts language model, achieves superior performance across benchmarks by using a selective architecture that leverages fewer active parameters.
paper_ai_keywords: Sparse Mixture of Experts (SMoE) feedforward blocks experts router network context size Llama 2 70B GPT-3.5 mathematics code generation multilingual benchmarks Mixtral 8x7B - Instruct human benchmarks