Recent efforts in natural language processing (NLP) have demonstrated promising advancements in
compressing transformer-based models. Knowledge distillation (KD) has successfully been applied
to reduce the size of models such as BERT (Devlin et al., 2019), without a significant performance
loss on non-generative classification tasks (Sanh et al., 2019; Jiao et al., 2020; Sun et al., 2020).
Inspired by machine translation methods, pseudo-labelling (PL) approaches (Kim & Rush, 2016)
have also been explored for Seq2Seq summarisation (Shleifer & Rush, 2020), demonstrating the
potential for substantial compression of Seq2Seq models on generative tasks. In the audio domain,
KD has shown promising results for audio classification (Peng et al., 2021; Chang et al., 2021).
However, similar results have not yet been achieved for the more difficult task of speech recognition.
In this paper, we apply distillation to the Whisper model in the context of Seq2Seq ASR. We address
DISTIL-WHISPER