10

Step80.082.585.087.590.092.595.097.5100.0MetricSuperGLUE CB TaskSparse train_evalSparse validation_evalDense train_evalDense validation_evalStep8486889092949698100102MetricSuperGLUE ReCoRD TaskSparse train_evalSparse validation_evalDense train_evalDense validation_eval0.00.10.20.3Dropout Probability80828486SuperGLUE Score0.00.10.20.30.40.5Expert Dropout Probability80828486SuperGLUE Score4.2 FINE-TUNING A SUBSET OF MODEL PARAMETERS TO IMPROVE GENERALIZATION

To combat overﬁtting we experiment updating only a subset of models parameters during ﬁne-
tuning. Figure 5 measures quality for updating 5 different subsets of parameters: all parameters
(All), only non MoE parameters (Non MoE), only MoE parameters (MoE), only the self-attention
and enc-dec attention parameters (Attention) and only the non MoE FFN parameters (FFN).
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS