Figure 7: ORT MoE performance improvement on the CLIP Vision model (NDV4 cluster)

ORT MoE integrates with Tutel computation optimization features and provides user an option to take
advantage of Tutel.

Distribution Strategies supported by ORT MoE
Unlike the traditional transformer structure (or neural network), the routing logic of MoE breaks the locality of
input tokens: instead of being only consumed by a single or a small set of GPUs (e.g., model parallelism), the
tokens on each GPU are routed globally to experts on other GPUs. This gives flexibility to how we distribute
the experts and provides opportunities to explore distribution strategies with different communication
patterns and the associated trade-offs. As different workloads may require different distributions for best
performance, it is important to support a rich set of distribution strategies. In this section we discuss some of
the distribution strategies we used with ORT MoE.
Scaling Speech, Language and Vision Models with Mixture of Experts Technique - Microsoft Community Hub