p(missing data | context) needs to be accurate for every context. Hence, scale and diversity are the
most crucial factors for building general-purpose models [Hoffmann et al., 2022, Aghajanyan et al.,
2023], as we see the state-of-the-art (SOTA) language and vision-language models are trained on
web-scale data with billions to hundreds of billions of parameters.
Despite the success of large-scale generative models in other areas, most speech models are still
trained on datasets at the scale of tens to hundreds of hours [Ren et al., 2021, Kim et al., 2020, 2021,
Voicebox-Text-GuidedMultilingual UniversalSpeechGenerationatScale