the speech recognition pipeline since it removes the need
for a separate inverse text normalization step in order to
produce naturalistic transcriptions.
We construct the dataset from audio that is paired with tran-
scripts on the Internet. This results in a very diverse dataset
covering a broad distribution of audio from many different
environments, recording setups, speakers, and languages.
While diversity in audio quality can help train a model to be
robust, diversity in transcript quality is not similarly bene-
ﬁcial. Initial inspection showed a large amount of subpar
transcripts in the raw dataset. To address this, we developed
several automated ﬁltering methods to improve transcript
quality.
Many transcripts on the internet are not actually human-
generated but the output of existing ASR systems. Recent
research has shown that training on datasets of mixed human
and machine-generated data can signiﬁcantly impair the per-
formance of translation systems (Ghorbani et al., 2021). In
RobustSpeechRecognitionviaLarge-ScaleWeakSupervision