Sparse expert models are sensitive to roundoff errors because they have more exponential
functions due to the routers. Sparse expert models introduce additional exponential functions –
through the router – which can exacerbate roundoff errors7 and lead to training instabilities. While
a roundoff error does not change the ordering of probabilities within a softmax operation, it does
impact the routing of the second token in MoE due to relative thresholding (e.g. a token is only
routed to its second place expert if the gating probability for the second expert is 1/5 as large as
that of the ﬁrst expert). Additionally, roundoff errors can drastically change the probability that
scales the expert output – which we have found to be important. Finally, we conjecture that the
higher stability we observed for decoder-only models (not shown here) was because they had fewer
exponential functions. Section 9 contains a more detailed discussion.
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS