Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive
  Bias
Scaling text-to-speech to a large and wild dataset has been proven to be
highly effective in achieving timbre and speech style generalization,
particularly in zero-shot TTS. However, previous works usually encode speech
into latent using audio codec and use autoregressive language models or
diffusion models to generate it, which ignores the intrinsic nature of speech
and may lead to inferior or uncontrollable results. We argue that speech can be
decomposed into several attributes (e.g., content, timbre, prosody, and phase)
and each of them should be modeled using a module with appropriate inductive
biases. From this perspective, we carefully design a novel and large zero-shot
TTS system called Mega-TTS, which is trained with large-scale wild data and
models different attributes in different ways: 1) Instead of using latent
encoded by audio codec as the intermediate feature, we still choose spectrogram
as it separates the phase and other attributes very well. Phase can be
appropriately constructed by the GAN-based vocoder and does not need to be
modeled by the language model. 2) We model the timbre using global vectors
since timbre is a global attribute that changes slowly over time. 3) We further
use a VQGAN-based acoustic model to generate the spectrogram and a latent code
language model to fit the distribution of prosody, since prosody changes
quickly over time in a sentence, and language models can capture both local and
long-range dependencies. We scale Mega-TTS to multi-domain datasets with 20K
hours of speech and evaluate its performance on unseen speakers. Experimental
results demonstrate that Mega-TTS surpasses state-of-the-art TTS systems on
zero-shot TTS, speech editing, and cross-lingual TTS tasks, with superior
naturalness, robustness, and speaker similarity due to the proper inductive
bias of each module. Audio samples are available at
https://mega-tts.github.io/demo-page.
647fdd3700bb4708d2496754
Ziyue Jiang
claimed_verified
2023-06-07T19:40:07.529Z
6268b9d887d7f9040e0787d3
/avatars/5d022d1e1d7f247e94928ef8ad04c613.svg
jiang
user
ziyue
647fdd3700bb4708d2496755
Yi Ren
647fdd3700bb4708d2496756
Zhenhui Ye
claimed_verified
2023-06-07T19:40:15.243Z
6277fcd8c5c547c31b80cc77
/avatars/7dab3f5c879b5c5eb86f822119a9b5bb.svg
Zhenhui Ye
user
yerfor
647fdd3700bb4708d2496757
Jinglin Liu
claimed_verified
2023-06-14T09:29:38.999Z
6205c6ad00e630c7a5d0fc2e
https://cdn-avatars.huggingface.co/v1/production/uploads/1645184905991-6205c6ad00e630c7a5d0fc2e.jpeg
Jinglin Liu
user
Silentlin
647fdd3700bb4708d2496758
Chen Zhang
647fdd3700bb4708d2496759
Qian Yang
647fdd3700bb4708d249675a
Shengpeng Ji
647fdd3700bb4708d249675b
Rongjie Huang
647fdd3700bb4708d249675c
Chunfeng Wang
647fdd3700bb4708d249675d
Xiang Yin
647fdd3700bb4708d249675e
Zejun Ma
647fdd3700bb4708d249675f
Zhou Zhao
Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive
  Bias
Scaling text-to-speech to a large and wild dataset has been proven to be
highly effective in achieving timbre and speech style generalization,
particularly in zero-shot TTS. However, previous works usually encode speech
into latent using audio codec and use autoregressive language models or
diffusion models to generate it, which ignores the intrinsic nature of speech
and may lead to inferior or uncontrollable results. We argue that speech can be
decomposed into several attributes (e.g., content, timbre, prosody, and phase)
and each of them should be modeled using a module with appropriate inductive
biases. From this perspective, we carefully design a novel and large zero-shot
TTS system called Mega-TTS, which is trained with large-scale wild data and
models different attributes in different ways: 1) Instead of using latent
encoded by audio codec as the intermediate feature, we still choose spectrogram
as it separates the phase and other attributes very well. Phase can be
appropriately constructed by the GAN-based vocoder and does not need to be
modeled by the language model. 2) We model the timbre using global vectors
since timbre is a global attribute that changes slowly over time. 3) We further
use a VQGAN-based acoustic model to generate the spectrogram and a latent code
language model to fit the distribution of prosody, since prosody changes
quickly over time in a sentence, and language models can capture both local and
long-range dependencies. We scale Mega-TTS to multi-domain datasets with 20K
hours of speech and evaluate its performance on unseen speakers. Experimental
results demonstrate that Mega-TTS surpasses state-of-the-art TTS systems on
zero-shot TTS, speech editing, and cross-lingual TTS tasks, with superior
naturalness, robustness, and speaker similarity due to the proper inductive
bias of each module. Audio samples are available at
https://mega-tts.github.io/demo-page.
Mega-TTS, a zero-shot TTS system, uses attribute-specific models trained on large datasets to generate high-quality speech with superior naturalness, robustness, and speaker similarity in zero-shot tasks.
diffusion models spectrogram GAN-based vocoder global vectors VQGAN-based acoustic model latent code language model zero-shot TTS speech editing cross-lingual TTS