title: Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech
    Prompts
summary: Zero-shot text-to-speech aims at synthesizing voices with unseen speech
  prompts. Previous large-scale multispeaker TTS models have successfully
  achieved this goal with an enrolled recording within 10 seconds. However, most
  of them are designed to utilize only short speech prompts. The limited
  information in short speech prompts significantly hinders the performance of
  fine-grained identity imitation. In this paper, we introduce Mega-TTS 2, a
  generic zero-shot multispeaker TTS model that is capable of synthesizing speech
  for unseen speakers with arbitrary-length prompts. Specifically, we 1) design a
  multi-reference timbre encoder to extract timbre information from multiple
  reference speeches; 2) and train a prosody language model with arbitrary-length
  speech prompts; With these designs, our model is suitable for prompts of
  different lengths, which extends the upper bound of speech quality for
  zero-shot text-to-speech. Besides arbitrary-length prompts, we introduce
  arbitrary-source prompts, which leverages the probabilities derived from
  multiple P-LLM outputs to produce expressive and controlled prosody.
  Furthermore, we propose a phoneme-level auto-regressive duration model to
  introduce in-context learning capabilities to duration modeling. Experiments
  demonstrate that our method could not only synthesize identity-preserving
  speech with a short prompt of an unseen speaker but also achieve improved
  performance with longer speech prompts. Audio samples can be found in
  https://mega-tts.github.io/mega2_demo/.
paper_authors._id: 64b4ac01a15d33a1bcc5f856
paper_authors.name: Ziyue Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-17T10:45:17.737Z
paper_authors._id: 6268b9d887d7f9040e0787d3
paper_authors.avatarUrl: /avatars/5d022d1e1d7f247e94928ef8ad04c613.svg
paper_authors.fullname: jiang
paper_authors.type: user
paper_authors.user: ziyue
paper_authors._id: 64b4ac01a15d33a1bcc5f857
paper_authors.name: Jinglin Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-17T10:45:36.306Z
paper_authors._id: 6205c6ad00e630c7a5d0fc2e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1645184905991-6205c6ad00e630c7a5d0fc2e.jpeg
paper_authors.fullname: Jinglin Liu
paper_authors.type: user
paper_authors.user: Silentlin
paper_authors._id: 64b4ac01a15d33a1bcc5f858
paper_authors.name: Yi Ren
paper_authors._id: 64b4ac01a15d33a1bcc5f859
paper_authors.name: Jinzheng He
paper_authors._id: 64b4ac01a15d33a1bcc5f85a
paper_authors.name: Chen Zhang
paper_authors._id: 64b4ac01a15d33a1bcc5f85b
paper_authors.name: Zhenhui Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-17T11:07:21.688Z
paper_authors._id: 6277fcd8c5c547c31b80cc77
paper_authors.avatarUrl: /avatars/7dab3f5c879b5c5eb86f822119a9b5bb.svg
paper_authors.fullname: Zhenhui Ye
paper_authors.type: user
paper_authors.user: yerfor
paper_authors._id: 64b4ac01a15d33a1bcc5f85c
paper_authors.name: Pengfei Wei
paper_authors._id: 64b4ac01a15d33a1bcc5f85d
paper_authors.name: Chunfeng Wang
paper_authors._id: 64b4ac01a15d33a1bcc5f85e
paper_authors.name: Xiang Yin
paper_authors._id: 64b4ac01a15d33a1bcc5f85f
paper_authors.name: Zejun Ma
paper_authors._id: 64b4ac01a15d33a1bcc5f860
paper_authors.name: Zhou Zhao
paper_title: Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech
    Prompts
paper_summary: Zero-shot text-to-speech aims at synthesizing voices with unseen speech
  prompts. Previous large-scale multispeaker TTS models have successfully
  achieved this goal with an enrolled recording within 10 seconds. However, most
  of them are designed to utilize only short speech prompts. The limited
  information in short speech prompts significantly hinders the performance of
  fine-grained identity imitation. In this paper, we introduce Mega-TTS 2, a
  generic zero-shot multispeaker TTS model that is capable of synthesizing speech
  for unseen speakers with arbitrary-length prompts. Specifically, we 1) design a
  multi-reference timbre encoder to extract timbre information from multiple
  reference speeches; 2) and train a prosody language model with arbitrary-length
  speech prompts; With these designs, our model is suitable for prompts of
  different lengths, which extends the upper bound of speech quality for
  zero-shot text-to-speech. Besides arbitrary-length prompts, we introduce
  arbitrary-source prompts, which leverages the probabilities derived from
  multiple P-LLM outputs to produce expressive and controlled prosody.
  Furthermore, we propose a phoneme-level auto-regressive duration model to
  introduce in-context learning capabilities to duration modeling. Experiments
  demonstrate that our method could not only synthesize identity-preserving
  speech with a short prompt of an unseen speaker but also achieve improved
  performance with longer speech prompts. Audio samples can be found in
  https://mega-tts.github.io/mega2_demo/.
paper_ai_keywords: zero-shot text-to-speech multispeaker TTS timbre encoder prosody language arbitrary-length prompts in-context learning duration modeling identity-preserving speech