examples. Instead, we swept over a set of values of the threshold, recording the WER performance
and number of decoder layers used.
Table 20 shows the average number of decoder layers utilised by the medium.en model as the early-
exit threshold is reduced. The medium.en model has a total of 24 decoder layers, of which the last
3 are skipped almost immediately. However, the WER penalty is significant even for just a 3-layer
reduction. As we reduce the threshold, the number of layers skipped does not reduce significantly,
but the WER penalty continues to inflate. Setting the threshold to 0.9750 results in an average of
3 skipped decoder layers, yielding an inference speed-up of 1.1 times. However, it also causes an
increase in WER from 2.3% to 3.4% This suggests that there is high-utilisation of the first 21 decoder
layers in the pre-trained Whisper model, and that the final 3 layers are necessary for ensuring high
DISTIL-WHISPER