unknown Meetings
18,260+

540 Telephone conversations

Figure 1: Architecture of the Distil-Whisper model. The encoder (shown in green) is entirely
copied from the teacher to the student and frozen during training. The student’s decoder consists
of only two decoder layers, which are initialised from the first and last decoder layer of the teacher
(shown in red). All other decoder layers of the teacher are discarded. The model is trained on a
weighted sum of the KL divergence and PL loss terms.

7.2 TRAINING
DISTIL-WHISPER