title: Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive
    Bias
summary: Scaling text-to-speech to a large and wild dataset has been proven to be
  highly effective in achieving timbre and speech style generalization,
  particularly in zero-shot TTS. However, previous works usually encode speech
  into latent using audio codec and use autoregressive language models or
  diffusion models to generate it, which ignores the intrinsic nature of speech
  and may lead to inferior or uncontrollable results. We argue that speech can be
  decomposed into several attributes (e.g., content, timbre, prosody, and phase)
  and each of them should be modeled using a module with appropriate inductive
  biases. From this perspective, we carefully design a novel and large zero-shot
  TTS system called Mega-TTS, which is trained with large-scale wild data and
  models different attributes in different ways: 1) Instead of using latent
  encoded by audio codec as the intermediate feature, we still choose spectrogram
  as it separates the phase and other attributes very well. Phase can be
  appropriately constructed by the GAN-based vocoder and does not need to be
  modeled by the language model. 2) We model the timbre using global vectors
  since timbre is a global attribute that changes slowly over time. 3) We further
  use a VQGAN-based acoustic model to generate the spectrogram and a latent code
  language model to fit the distribution of prosody, since prosody changes
  quickly over time in a sentence, and language models can capture both local and
  long-range dependencies. We scale Mega-TTS to multi-domain datasets with 20K
  hours of speech and evaluate its performance on unseen speakers. Experimental
  results demonstrate that Mega-TTS surpasses state-of-the-art TTS systems on
  zero-shot TTS, speech editing, and cross-lingual TTS tasks, with superior
  naturalness, robustness, and speaker similarity due to the proper inductive
  bias of each module. Audio samples are available at
  https://mega-tts.github.io/demo-page.
paper_authors._id: 647fdd3700bb4708d2496754
paper_authors.name: Ziyue Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-07T19:40:07.529Z
paper_authors._id: 6268b9d887d7f9040e0787d3
paper_authors.avatarUrl: /avatars/5d022d1e1d7f247e94928ef8ad04c613.svg
paper_authors.fullname: jiang
paper_authors.type: user
paper_authors.user: ziyue
paper_authors._id: 647fdd3700bb4708d2496755
paper_authors.name: Yi Ren
paper_authors._id: 647fdd3700bb4708d2496756
paper_authors.name: Zhenhui Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-07T19:40:15.243Z
paper_authors._id: 6277fcd8c5c547c31b80cc77
paper_authors.avatarUrl: /avatars/7dab3f5c879b5c5eb86f822119a9b5bb.svg
paper_authors.fullname: Zhenhui Ye
paper_authors.type: user
paper_authors.user: yerfor
paper_authors._id: 647fdd3700bb4708d2496757
paper_authors.name: Jinglin Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-14T09:29:38.999Z
paper_authors._id: 6205c6ad00e630c7a5d0fc2e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1645184905991-6205c6ad00e630c7a5d0fc2e.jpeg
paper_authors.fullname: Jinglin Liu
paper_authors.type: user
paper_authors.user: Silentlin
paper_authors._id: 647fdd3700bb4708d2496758
paper_authors.name: Chen Zhang
paper_authors._id: 647fdd3700bb4708d2496759
paper_authors.name: Qian Yang
paper_authors._id: 647fdd3700bb4708d249675a
paper_authors.name: Shengpeng Ji
paper_authors._id: 647fdd3700bb4708d249675b
paper_authors.name: Rongjie Huang
paper_authors._id: 647fdd3700bb4708d249675c
paper_authors.name: Chunfeng Wang
paper_authors._id: 647fdd3700bb4708d249675d
paper_authors.name: Xiang Yin
paper_authors._id: 647fdd3700bb4708d249675e
paper_authors.name: Zejun Ma
paper_authors._id: 647fdd3700bb4708d249675f
paper_authors.name: Zhou Zhao
paper_title: Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive
    Bias
paper_summary: Scaling text-to-speech to a large and wild dataset has been proven to be
  highly effective in achieving timbre and speech style generalization,
  particularly in zero-shot TTS. However, previous works usually encode speech
  into latent using audio codec and use autoregressive language models or
  diffusion models to generate it, which ignores the intrinsic nature of speech
  and may lead to inferior or uncontrollable results. We argue that speech can be
  decomposed into several attributes (e.g., content, timbre, prosody, and phase)
  and each of them should be modeled using a module with appropriate inductive
  biases. From this perspective, we carefully design a novel and large zero-shot
  TTS system called Mega-TTS, which is trained with large-scale wild data and
  models different attributes in different ways: 1) Instead of using latent
  encoded by audio codec as the intermediate feature, we still choose spectrogram
  as it separates the phase and other attributes very well. Phase can be
  appropriately constructed by the GAN-based vocoder and does not need to be
  modeled by the language model. 2) We model the timbre using global vectors
  since timbre is a global attribute that changes slowly over time. 3) We further
  use a VQGAN-based acoustic model to generate the spectrogram and a latent code
  language model to fit the distribution of prosody, since prosody changes
  quickly over time in a sentence, and language models can capture both local and
  long-range dependencies. We scale Mega-TTS to multi-domain datasets with 20K
  hours of speech and evaluate its performance on unseen speakers. Experimental
  results demonstrate that Mega-TTS surpasses state-of-the-art TTS systems on
  zero-shot TTS, speech editing, and cross-lingual TTS tasks, with superior
  naturalness, robustness, and speaker similarity due to the proper inductive
  bias of each module. Audio samples are available at
  https://mega-tts.github.io/demo-page.
paper_ai_keywords: pre-trained TTS audio codec autoregressive language models diffusion models spectrogram GAN-based vocoder global vectors VQGAN-based acoustic model latent code language model zero-shot TTS speech editing cross-lingual TTS