We study the translation capabilities of Whisper models
by measuring their performance on the X→en subset of
CoVoST2 (Wang et al., 2020b). We compare with Maestro,
mSLAM, and XLS-R, the highest-performing prior work.
We achieve a new state of the art of 29.1 BLEU zero-shot
without using any of the CoVoST2 training data. We at-
tribute this to the 68,000 hours of X→en translation data
for these languages in our pre-training dataset which, al-
though noisy, is vastly larger than the 861 hours of training
data for X→en translation in CoVoST2. Since Whisper eval-
uation is zero-shot, it does particularly well on the lowest
resource grouping of CoVoST2, improving over mSLAM
by 6.7 BLEU. Conversely, the best Whisper model does not
actually improve over Maestro and mSLAM on average for
the highest resource languages.
For an additional analysis on an even wider set of languages,
we also re-purpose Fleurs, which is a speech recognition
dataset, as a translation dataset. Since the same sentences
RobustSpeechRecognitionviaLarge-ScaleWeakSupervision