Detailed results for Mixtral, Mistral 7B and Llama 2 7B/13B/70B and Llama 1 34B2 are reported
in Table 2. Figure 2 compares the performance of Mixtral with the Llama models in different
categories. Mixtral surpasses Llama 2 70B across most metrics. In particular, Mixtral displays a
superior performance in code and mathematics benchmarks.
Size and Efficiency. We compare our performance to the Llama 2 family, aiming to understand
Mixtral models’ efficiency in the cost-performance spectrum (see Figure 3). As a sparse Mixture-
of-Experts model, Mixtral only uses 13B active parameters for each token. With 5x lower active
parameters, Mixtral is able to outperform Llama 2 70B across most categories.
Note that this analysis focuses on the active parameter count (see Section 2.1), which is directly
proportional to the inference compute cost, but does not consider the memory costs and hardware
utilization. The memory costs for serving Mixtral are proportional to its sparse parameter count,
Mixtral of Experts paper