This technique increases the number of parameters of a model while controlling cost and latency, as the
model only uses a fraction of the total set of parameters per token. Concretely, Mixtral has 46.7B total
parameters  but  only  uses  12.9B  parameters  per  token.  It,  therefore,  processes  input  and  generates
output at the same speed and for the same cost as a 12.9B model.

Mixtral  is  pre-trained  on  data  extracted  from  the  open  Web  –  we  train  experts  and  routers
simultaneously.

Performance
We compare Mixtral to the Llama 2 family and the GPT3.5 base model. Mixtral matches or outperforms
Llama 2 70B, as well as GPT3.5, on most benchmarks.

On the following figure, we measure the quality versus inference budget tradeoff. Mistral 7B and Mixtral
8
Mixtral of experts