replacement for base.en at low batch sizes and small.en at higher ones, with 3.3% and 0.7% WER
improvements respectively. Similarly, distil-medium.en is faster than tiny.en at batch sizes 1 and 4,
and faster than base.en at batch size 16, while performing 3.2% better on OOD test data.
Incorporating FA2 benefits Distil-Whisper more than Whisper at higher batch sizes: the distil-large-
v2 model is 31% faster with FA2 at batch size 16, compared to 22% for large-v2. Furthermore,
the distil-medium.en checkpoint is as fast as tiny.en at batch size 16, with an average OOD WER
that is 7.8% lower. FA2 has a greater improvement on the encoder than the decoder, where the
memory is re-allocated at each decoding step. Since Distil-Whisper consists of 32 encoder layers
and only 2 decoder layers, it improves the inference time significantly more than Whisper, which
has 32 encoder and 32 decoder layers. This suggests that FA2 should always be incorporated for
DISTIL-WHISPER