potentially hurting efﬁciency. Increasing the number of experts does not change the computation
done (sparse models apply a ﬁxed amount of computation to each input), but increases the mem-
ory transfer requirement (additional expert variables must be loaded from device memory). This
decreases the compute-to-memory ratio9.
On our TPU system, we recommend to one expert (or less) per core. Our largest models use both data
and model parallelism where data parallelism is over “rows” and model-parallelism over “columns”
of the logical mesh. We use ≤ 1 expert per data parallelism row to ensure the compute-to-memory
ratio is high and to reduce the cores needed for evaluation and inference. Furthermore, using less
experts lets us allocate more cores to the model parallelism “column” to have more FLOPs in our
model. Appendix H explains our mesh layouts for when we have fewer experts than data parallelism
rows.
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS