Mixtral of Experts
We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model.
Mixtral has the same architecture as Mistral 7B, with the difference that each
layer is composed of 8 feedforward blocks (i.e. experts). For every token, at
each layer, a router network selects two experts to process the current state
and combine their outputs. Even though each token only sees two experts, the
selected experts can be different at each timestep. As a result, each token has
access to 47B parameters, but only uses 13B active parameters during inference.
Mixtral was trained with a context size of 32k tokens and it outperforms or
matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular,
Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and
multilingual benchmarks. We also provide a model fine-tuned to follow
instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo,
Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both
the base and instruct models are released under the Apache 2.0 license.
659cb2f4e942a8f717b442ac
Albert Q. Jiang
admin_assigned
2024-01-09T07:56:34.904Z
6137dff56ecbbda5c86755e9
/avatars/1aadefd3acb7ad24df15463dadceb7d4.svg
Albert Q Jiang
user
aqj213
659cb2f4e942a8f717b442ad
Alexandre Sablayrolles
admin_assigned
2024-01-09T07:58:15.229Z
6391c7456176fbc67b9ecd2c
/avatars/4aa9be94d8284118b8018362abf11f01.svg
Alexandre Sablayrolles
user
alexsablay
659cb2f4e942a8f717b442ae
Antoine Roux
admin_assigned
2024-01-09T07:58:22.438Z
61643ec44175c9b7c811358a
/avatars/ef445c8c2605ce8ed48b0cd228a2b0b7.svg
Antoine Roux
user
antwanroux
659cb2f4e942a8f717b442af
Arthur Mensch
admin_assigned
2024-01-09T07:58:28.516Z
645d28525ebf379fd6d9ae41
https://cdn-avatars.huggingface.co/v1/production/uploads/645d28525ebf379fd6d9ae41/qVEKQk4gPlumsZE3ZorWG.jpeg
Arthur Mensch
user
arthurmensch
659cb2f4e942a8f717b442b0
Blanche Savary
659cb2f4e942a8f717b442b1
Chris Bamford
admin_assigned
2024-01-09T07:58:42.734Z
6304b9a2bad6ce7fc026cab6
https://cdn-avatars.huggingface.co/v1/production/uploads/1661254040723-noauth.png
Chris Bamford
user
Bam4d
659cb2f4e942a8f717b442b2
Devendra Singh Chaplot
admin_assigned
2024-01-09T07:58:49.031Z
65143e1c4f08b815c8db57a0
https://cdn-avatars.huggingface.co/v1/production/uploads/65143e1c4f08b815c8db57a0/JqkwKiJmLFRkH0NK3L8XH.jpeg
Devendra Singh Chaplot
user
devendrachaplot
659cb2f4e942a8f717b442b3
Diego de las Casas
admin_assigned
2024-01-09T07:58:55.362Z
657608c427c0e2fd6c894ced
/avatars/7f1ca5f75ca03cec69560584496558c9.svg
Diego de las Casas
user
diegolascasas
659cb2f4e942a8f717b442b4
Emma Bou Hanna
admin_assigned
2024-01-09T07:59:37.471Z
655dc9138e56f09d55197cac
/avatars/764edb5c5e1003eb71cf6f1bea563f4b.svg
Emma Bou Hanna
user
EmmaBH
659cb2f4e942a8f717b442b5
Florian Bressand
659cb2f4e942a8f717b442b6
Gianna Lengyel
659cb2f4e942a8f717b442b7
Guillaume Bour
659cb2f4e942a8f717b442b8
Guillaume Lample
admin_assigned
2024-01-09T08:00:40.393Z
64882fc9508d0656d43dd5b3
/avatars/c3bc3656e4ff05663b54c6dd4f241891.svg
Guillaume Lample
user
glample
659cb2f4e942a8f717b442b9
Lélio Renard Lavaud
659cb2f4e942a8f717b442ba
Lucile Saulnier
admin_assigned
2024-01-09T08:01:14.405Z
650cb0ebc705442b280160c4
/avatars/c5abfcddfa66505085178fa430779ed0.svg
Lucile Saulnier
user
LucileSaulnier
659cb2f4e942a8f717b442bb
Marie-Anne Lachaux
admin_assigned
2024-01-09T08:01:26.578Z
651e7ab594de702b4aa8ae51
/avatars/f097d3ca30e62567e600e85ca50e9a9d.svg
marie-anne lachaux
user
malachaux
659cb2f4e942a8f717b442bc
Pierre Stock
admin_assigned
2024-01-09T08:01:33.210Z
650c993d2751c84306aba92b
https://cdn-avatars.huggingface.co/v1/production/uploads/650c993d2751c84306aba92b/uS04hveXAu6CEszCy-8_G.jpeg
Pierre Stock
user
pstock
659cb2f4e942a8f717b442bd
Sandeep Subramanian
admin_assigned
2024-01-09T08:01:40.422Z
627bf27cf19c5eb46d54cea8
/avatars/b8ca0b4e841858c1d234671187234f56.svg
Sandeep Subramanian
user
MaximumEntropy
659cb2f4e942a8f717b442be
Sophia Yang
admin_assigned
2024-01-09T08:03:00.989Z
6320c05a145cfa4c04cb4359
https://cdn-avatars.huggingface.co/v1/production/uploads/6320c05a145cfa4c04cb4359/jLYLrlc_LZQMi3yCrlfCi.jpeg
Sophia Yang
user
sophiamyang
659cb2f4e942a8f717b442bf
Szymon Antoniak
admin_assigned
2024-01-09T08:03:13.031Z
63f767c4bd28622c9b984637
/avatars/126a23ca5b3adc89418b7a53b21e50e6.svg
Szymon Antoniak
user
Simontwice
659cb2f4e942a8f717b442c0
Teven Le Scao
admin_assigned
2024-01-09T08:03:20.358Z
5e67bed6100906368940747b
https://cdn-avatars.huggingface.co/v1/production/uploads/1583858339980-5e67bed6100906368940747b.jpeg
Teven Le Scao
user
teven
659cb2f4e942a8f717b442c1
Théophile Gervet
659cb2f4e942a8f717b442c2
Thibaut Lavril
659cb2f4e942a8f717b442c3
Thomas Wang
659cb2f4e942a8f717b442c4
Timothée Lacroix
admin_assigned
2024-01-15T09:53:10.733Z
6503364ceda77aafd4a68a48
https://cdn-avatars.huggingface.co/v1/production/uploads/6503364ceda77aafd4a68a48/POQqon4zWQEpFTPzHS4uu.png
timothee lacroix
user
timlacroix
659cb2f4e942a8f717b442c5
William El Sayed
Mixtral of Experts
We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model.
Mixtral has the same architecture as Mistral 7B, with the difference that each
layer is composed of 8 feedforward blocks (i.e. experts). For every token, at
each layer, a router network selects two experts to process the current state
and combine their outputs. Even though each token only sees two experts, the
selected experts can be different at each timestep. As a result, each token has
access to 47B parameters, but only uses 13B active parameters during inference.
Mixtral was trained with a context size of 32k tokens and it outperforms or
matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular,
Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and
multilingual benchmarks. We also provide a model fine-tuned to follow
instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo,
Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both
the base and instruct models are released under the Apache 2.0 license.
Mixtral 8x7B, a Sparse Mixture of Experts language model, achieves superior performance across benchmarks by using a selective architecture that leverages fewer active parameters.
Sparse Mixture of Experts (SMoE) feedforward blocks experts router network context size Llama 2 70B GPT-3.5 mathematics code generation multilingual benchmarks Mixtral 8x7B - Instruct human benchmarks