2021] for speech continuation without using text. HuBERT units encode mostly content [Polyak
et al., 2021], and the generated speech does not preserve the voice of the prompt. To tackle this,
AudioLM [Borsos et al., 2022a] considers a cascaded approach which first generates HuBERT-like
tokens and then predicts SoundStream [Zeghidour et al., 2022] tokens, a reconstruction based codec
that preserves style. These models are not conditioned on text and are evaluated on spoken language
modeling tasks.
VALL-E [Wang et al., 2023] is most related to Voicebox. It is a text conditioned LM trained on
Encodec [Défossez et al., 2022] tokens (similar to SoundStream). Encodec tokenizes speech with a
residual quantization layer, which encodes each frame with 8 codebooks at a 75Hz frame rate. The
codebooks are ordered such that the first code contains the most information and so on. VALL-E has
two modules. The first is an auto-regressive (AR) model that predicts the first code of each frame
Voicebox-Text-GuidedMultilingual UniversalSpeechGenerationatScale