[35] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai,
Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in
Neural Information Processing Systems, 35:7103–7114, 2022.

11

Figure 9: Proportion of tokens assigned to each expert on different subsets from The Pile dataset, separated
by whether the expert was selected as first or second choice, or either. The “Either choice” case is equivalent
to Figure 7. The gray dashed vertical line marks 1

8 , i.e. the proportion expected with uniform sampling.

12
Mixtral of Experts paper