7.2 TRAINING

We initialise the student models by copying the entire encoder from the teacher and freeze it dur-
ing training. We distill 2-layer decoder checkpoints from the medium.en and large-v2 models by
copying the first and last decoder layers, which we refer to as distil-medium.en and distil-large-
v2 respectively. The dimensionality details of the distilled models are shown in Table 3, with the
architecture and training objective summarised in Figure 1.
We train with a batch size of 256 for a total of 80,000 optimisation steps, which amounts to eight
epochs of training. Since we only train for eight epochs, the risk of over-fitting is low, and so we do
not use any data augmentation or regularisation techniques. Instead, we rely on the diversity of our
dataset to ensure model generalisation and robustness, the same premise used in training the original
Whisper model (Radford et al., 2022). Refer to Appendix B.1 for full details of our training set-up.

6
DISTIL-WHISPER