Deﬁnition
An independently-learned neural network with unique weights.
A network that computes the probability of each token getting sent to each
expert.
Routing algorithm where each token is routed to n experts.
An auxiliary (aux) loss to encourage each group of tokens to evenly distribute
across experts.
The global batch size is split into smaller groups, each of size Group Size. Each
group is considered separately for load balancing across experts. Increasing it
increases memory, computation, and communication.
Each expert can only process up to a ﬁxed number of tokens, which is often
set by evenly dividing across experts, tokens
experts . The capacity factor can expand or
contract this amount to CF · tokens
experts .
Acronym of Feed Forward Network (FFN) layer of Transformer consisting of
linear, activation, linear.
A Transformer architectural variant that all of our models are based on. Consists
of an encoder that does all-to-all attention on the inputs and a decoder that
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS