Figure 7: Proportion of tokens assigned to each expert on different domains from The Pile dataset for
layers 0, 15, and 31. The gray dashed vertical line marks 1/8, i.e. the proportion expected with uniform
sampling. Here, we consider experts that are either selected as a first or second choice by the router. A
breakdown of the proportion of assignments done in each case cane be seen in Figure 9 in the Appendix.

7

00.050.100.150.20layer: 000.050.100.150.20layer: 150123456700.050.100.150.20layer: 31Expert IDSelection proportionArXivDM MathematicsGithubGutenbergPhilPapersPubMed AbstractsStackExchangeWikipedia (en)ArXiv
DM Mathematics
Github
Gutenberg
PhilPapers
PubMed Abstracts
StackExchange
Wikipedia (en)

Layer 0
14.0%
14.1%
14.9%
13.9%
13.6%
14.2%
13.6%
14.4%

First choice
Layer 15
27.9%
28.4%
28.1%
26.1%
25.3%
24.6%
27.2%
23.6%

Layer 31
22.7%
19.7%
19.7%
26.3%
22.1%
22.0%
23.6%
25.3%

First or second choice

Layer 0
46.5%
44.9%
49.9%
49.5%
46.9%
48.6%
48.2%
49.8%
Mixtral of Experts paper