Expert/Gating Freeze.
In an effort to enhance the generalization capabilities of sparse models and
combat overfitting, researchers have discovered that finetuning a subset of model parameters results
in improved generalization performance for ST-MoE models, as noted in the study by ST-MoE [56].
Interestingly, it was observed that updating non-MoE parameters yields similar outcomes to updating
all parameters, while updating only expert parameters performs slightly better.
We conducted experiments by freezing the gating function, expert modules, and MoE parameters
of the given model, as presented in Table 2. The results indicate that freezing either the expert or
MoE components negatively impacts performance. Conversely, freezing the gate slightly improves
performance, albeit not significantly. We postulate that this observation is related to the under-fitting
of the FLAN-MOE, as in Figure 5, which depicts the finetuning data efficiency ablation study.
Mixture-of-Experts