MoE Acceleration Scenarios
We discuss scenarios where MoE improves scalability and model quality for large scale Transformer based
models within Microsoft below.

Improving quality of Automatic Speech Recognition model using MoE
We are able to increase the quality of the Automatic Speech Recognition model by 16.3% by applying the
MoE technique. We have used ORT MoE implementation to introduce experts layer in transformer networks
for multi-lingual speech recognition tasks. To train the multi-lingual network, we used approximately 75,000
hours of data from 10 languages. The baseline network for the model consists of a Transformer encoder with
18 blocks and a decoder with 6 blocks, where each block contains 8 heads of the attention network followed
by the feed-forwarding network (FFN) with pre-normalization. MoE is applied to every 2 FFN layers; see here
for details.
Scaling Speech, Language and Vision Models with Mixture of Experts Technique - Microsoft Community Hub