class classification and is proposed in this paper. In ORT MoE, it is a variant of the original z-loss
following Google’s TensorFlow MoE. The `ideal_load_balancing_loss` is a straight through estimator of
the ideal load balance which is not differentiable.
Gating Utilities: ORT MoE also provides gating utilities for balancing exploration. For top-1 gating,
`switch_jitter` will add multiplicative jitter noise on the incoming representations. `switch_dropout` will
add input dropout on the incoming representations. For both top-1 and top-2 gating, `logits_gumbel`
uses Gumbel-max trick to sample from the softmax distribution. `token_drop_type` includes 3 methods
for sample dropping when the expert capacity is not enough: `cut` means cut the sequence tail,
`random` means random sampling, and `routing_weight` means dropping samples according to the
routing weights. For top-2 gating, a 2nd-place loss can be added by specifying the ratio using `
second_place_loss_ratio`.
Scaling Speech, Language and Vision Models with Mixture of Experts Technique - Microsoft Community Hub