utilization. The memory costs for serving Mixtral are proportional to its sparse parameter count,
47B, which is still smaller than Llama 2 70B. As for device utilization, we note that the SMoEs layer
introduces additional overhead due to the routing mechanism and due to the increased memory loads
when running more than one expert per device. They are more suitable for batched workloads where
one can reach a good degree of arithmetic intensity.
Comparison with Llama 2 70B and GPT-3.5. In Table 3, we report the performance of Mixtral 8x7B
compared to Llama 2 70B and GPT-3.5. We observe that Mixtral performs similarly or above the
two other models. On MMLU, Mixtral obtains a better performance, despite its significantly smaller
capacity (47B tokens compared to 70B). For MT Bench, we report the performance of the latest
GPT-3.5-Turbo model available, gpt-3.5-turbo-1106.
Mixtral of Experts paper