Figure 1: Mixture of Experts Layer. Each input vector is assigned to 2 of the 8 experts by a router. The
layer’s output is the weighted sum of the outputs of the two selected experts. In Mixtral, an expert is a standard
feedforward block as in a vanilla transformer architecture.
Mixtral demonstrates superior capabilities in mathematics, code generation, and tasks that require
multilingual understanding, significantly outperforming Llama 2 70B in these domains. Experiments
show that Mixtral is able to successfully retrieve information from its context window of 32k tokens,
regardless of the sequence length and the location of the information in the sequence.
We also present Mixtral 8x7B – Instruct, a chat model fine-tuned to follow instructions using
supervised fine-tuning and Direct Preference Optimization [25]. Its performance notably surpasses
that of GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B – chat model on human evaluation
Mixtral of Experts paper