⋯⋯2 × Conv1D + GELU⋮cross attentionLog-Mel Spectrogram~SOTENTRANS- CRIBE0.0ThequickTokens in Multitask Training FormatTransformer Encoder Blocks Transformer Decoder Blocks EN0.0Thequickbrown⋮⋮next-token predictionSinusoidal Positional EncodingLearned Positional EncodingMultitask training data (680k hours)Sequence-to-sequence learningMultitask training formatEnglish transcriptionAny-to-English speech translationNon-English transcriptionNo speech🗣   “Ask not what your country can do for ⋯” 📝  Ask not what your country can do for ⋯🗣   “El rápido zorro marrón salta sobre ⋯” 📝  The quick brown fox jumps over ⋯🗣  “언덕 위에 올라 내려다보면 너무나 넓고 넓은 ⋯” 📝  언덕 위에 올라 내려다보면 너무나 넓고 넓은 ⋯🔊 (background music playing) 📝  ∅PREVspecialtokenstext tokenstimestamptokensSTART OF TRANSCRIPTLANGUAGE TAGNO SPEECHEOTTRANSCRIBETRANSLATEbegin timeNO TIMESTAMPS⋯end timetext tokensbegin timeend timetext tokenstext tokensVoice activity detection (VAD)Custom vocabulary /promptingTime-aligned transcriptionText-only
RobustSpeechRecognitionviaLarge-ScaleWeakSupervision