We have also investigated if the MoE teacher model can be compressed without loss of recognition accuracy.
For this study, we have developed Teacher-Student (TS) training with Knowledge Distillation (KD). To make the
student model closer to the teacher model, we initialize the student model with teacher’s weights. For
initialization of student’s FFN, we select one of the teacher’s experts based on the frequency used in inference
on a validation dataset. The student model is updated with KD loss and then fine-tuned with supervised
learning. The details are described here. The table shows the WERs obtained with two teacher models with 24
and 72 experts as well as the student model distilled from knowledge of each teacher model. As a baseline,
the table also shows the WER of the student model trained from scratch without the teacher model. It is clear
from the table that TS training can provide better accuracy than the model trained from scratch whether a
Scaling Speech, Language and Vision Models with Mixture of Experts Technique - Microsoft Community Hub