Experts are at the heart of the Mixture of Experts technique. Usually, a standard feedforward neural network
sublayer is used as an expert but that is not necessary. Fundamentally, the Mixture of Experts technique
deploys many experts to increase model parameter count to achieve better model quality. However, at
runtime only a small subset of these experts is used to process the given input tokens. This allows data
scientists to keep FLOP budget under control while increasing the model size. A gating function is used to
select a small subset of experts at runtime. Top1 or Top2 algorithms are popular choices as gating functions.
MixtureOfExperts sublayer consists of the gating function, experts and the needed communication collectives
to synchronize experts across multiple shards. This sublayer is the core of an MoE implementation. Finally, a
Transformer layer is constructed using MixtureOfExperts sub layer and other components such as multi-
headed attention layer.
Scaling Speech, Language and Vision Models with Mixture of Experts Technique - Microsoft Community Hub