Zero-shot TTS & alignment-preserved style transfer Given a target text ˆy and a transcribed
reference audio (x, y), zero-shot TTS aims to synthesize speech resembling the possibly unseen
audio style of the reference. Voicebox performs the task by treating the reference audio and the
target speech as one utterance where the target speech is masked. Let l and z be phone duration and
frame-level transcript of (x, y). The target duration ˆl is sampled given the duration context l and
concatenated phone sequence cat(y, ˆy). The target speech ˆx is then sampled given the context x and
concatenated frame-level phones cat(z, rep(ˆy, ˆl)).
Voicebox can also convert the audio style for speech ¯x while preserving its alignment ¯z. This is
useful for editing audio that is synchronized with other modalities such as video. Similar to zero-shot
TTS, Voicebox can simply perform the task by sampling target speech ˆx given the context x and
concatenated frame-level phones cat(z, ¯z).
Voicebox-Text-GuidedMultilingual UniversalSpeechGenerationatScale