7Exponential functions have the property that a small input perturbation can lead to a large difference in
the output. As an example, consider inputting 10 logits to a softmax function with values of 128 and one logit
with a value 128.5. A roundoff error of 0.5 in bfloat16 will alter the softmax output by 36% and incorrectly
exp(0)+10·exp(0) ≈ 0.091.
make all logits equal. The calculation goes from
This occurs because the max is subtracted from all logits (for numerical stability) in softmax operations and
the roundoff error changes the number from 128.5 to 128. This example was in bfloat16, but analogous
situations occur in float32 with larger logit values.

exp(0)+10·exp(−0.5) ≈ 0.142 to

exp(0)

exp(0)

9
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS