Expert Parallelism (EP):
In this scheme, experts are evenly divided across ranks. Alltoall collectives are inserted before and after
experts to route the input and output tokens to the correct position, according to the gating function. The
non-expert parameters are replicated across ranks and synchronized using AllReduction collective. This is the
distribution strategy that was introduced in the GShard paper.

This distribution enables a simple and good load-balanced distribution of MoE and has been widely used in
different models. In this distribution, the performance of Alltoall is one critical factor of the throughput.

 

 

 

Figure 8: Expert Parallelism as described in Gshard paper
Scaling Speech, Language and Vision Models with Mixture of Experts Technique - Microsoft Community Hub