4.2. Dataset Scaling

At 680,000 hours of labeled audio, the Whisper dataset is
one of the largest ever created in supervised speech recog-
nition. Exactly how important is the raw dataset size to
Whisper’s performance? To study this, we trained a series
of medium-sized models on subsampled versions of the
dataset which are 0.5%, 1%, 2%, 4%, and 8% of the full
dataset size and compared their performance with the same
medium-sized model trained on the whole dataset. Early
stopping based on the validation loss was used to select
model checkpoints for each dataset size. Evaluation was
performed on an exponential moving average estimate of
the parameters (Polyak & Juditsky, 1992) using a smooth-
ing rate of 0.9999 to help reduce the effect of the learning
rate not fully decaying to zero for the models trained on the
subsampled datasets due to early stopping. Performance
on English and multilingual speech recognition and X→en
translation is reported in Table 6.
RobustSpeechRecognitionviaLarge-ScaleWeakSupervision