Inspired by SpeechStew (Chan et al., 2021), we assemble a large corpus of ASR training data for
large-scale KD through a combination of nine publicly available speech recognition datasets. An
overview of the datasets is presented in Table 2, with additional details in Appendix A.1. The
combined dataset contains 21,170 hours of speech data, encompassing over 18,260 speakers and 10
distinct domains. We load and pre-process all datasets in the Hugging Face Datasets library (Lhoest
et al., 2021), streaming the data from the Hugging Face Hub1.
We generate pseudo-labels for our training data with the Whisper large-v2 checkpoint, using the
Flax Whisper implementation in the Hugging Face Transformers library (Heek et al., 2020; Wolf
et al., 2020). We found there to be little difference in the downstream performance of the distilled
model after pseudo-labelling using either greedy or beam-search, and so we opted to pseudo-label
DISTIL-WHISPER