Introduction
Mixture of Experts technique, frequently referred to as MoE, is gaining traction in the NLP community. It is
primarily used to scale Transformer models without incurring high computational resource costs. In this post,
we discuss how ORT MoE, an MoE implementation from the ONNX Runtime team, is used to scale networks
and improve the quality in Speech and Vision models in addition to NLP models. Automatic Speech
Recognition model is able to increase quality by 16.3% and language models are able to improve sample
efficiency by 10x using this technique. We will briefly describe MoE architecture first. ORT MoE features and
implementation details are discussed in the second half of this post.

The Mixture of Expert techniques typically use subcomponents such as Experts and, Gating Functions in a
Transformer block as shown in Figure 1.

 

Figure 1: MoE components
Scaling Speech, Language and Vision Models with Mixture of Experts Technique - Microsoft Community Hub