Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech
  Prompts
Zero-shot text-to-speech aims at synthesizing voices with unseen speech
prompts. Previous large-scale multispeaker TTS models have successfully
achieved this goal with an enrolled recording within 10 seconds. However, most
of them are designed to utilize only short speech prompts. The limited
information in short speech prompts significantly hinders the performance of
fine-grained identity imitation. In this paper, we introduce Mega-TTS 2, a
generic zero-shot multispeaker TTS model that is capable of synthesizing speech
for unseen speakers with arbitrary-length prompts. Specifically, we 1) design a
multi-reference timbre encoder to extract timbre information from multiple
reference speeches; 2) and train a prosody language model with arbitrary-length
speech prompts; With these designs, our model is suitable for prompts of
different lengths, which extends the upper bound of speech quality for
zero-shot text-to-speech. Besides arbitrary-length prompts, we introduce
arbitrary-source prompts, which leverages the probabilities derived from
multiple P-LLM outputs to produce expressive and controlled prosody.
Furthermore, we propose a phoneme-level auto-regressive duration model to
introduce in-context learning capabilities to duration modeling. Experiments
demonstrate that our method could not only synthesize identity-preserving
speech with a short prompt of an unseen speaker but also achieve improved
performance with longer speech prompts. Audio samples can be found in
https://mega-tts.github.io/mega2_demo/.
64b4ac01a15d33a1bcc5f856
Ziyue Jiang
admin_assigned
2023-07-17T10:45:17.737Z
6268b9d887d7f9040e0787d3
/avatars/5d022d1e1d7f247e94928ef8ad04c613.svg
jiang
user
ziyue
64b4ac01a15d33a1bcc5f857
Jinglin Liu
admin_assigned
2023-07-17T10:45:36.306Z
6205c6ad00e630c7a5d0fc2e
https://cdn-avatars.huggingface.co/v1/production/uploads/1645184905991-6205c6ad00e630c7a5d0fc2e.jpeg
Jinglin Liu
user
Silentlin
64b4ac01a15d33a1bcc5f858
Yi Ren
64b4ac01a15d33a1bcc5f859
Jinzheng He
64b4ac01a15d33a1bcc5f85a
Chen Zhang
64b4ac01a15d33a1bcc5f85b
Zhenhui Ye
admin_assigned
2023-07-17T11:07:21.688Z
6277fcd8c5c547c31b80cc77
/avatars/7dab3f5c879b5c5eb86f822119a9b5bb.svg
Zhenhui Ye
user
yerfor
64b4ac01a15d33a1bcc5f85c
Pengfei Wei
64b4ac01a15d33a1bcc5f85d
Chunfeng Wang
64b4ac01a15d33a1bcc5f85e
Xiang Yin
64b4ac01a15d33a1bcc5f85f
Zejun Ma
64b4ac01a15d33a1bcc5f860
Zhou Zhao
Mega-TTS 2: Zero-Shot Text-to-Speech with Arbitrary Length Speech
  Prompts
Zero-shot text-to-speech aims at synthesizing voices with unseen speech
prompts. Previous large-scale multispeaker TTS models have successfully
achieved this goal with an enrolled recording within 10 seconds. However, most
of them are designed to utilize only short speech prompts. The limited
information in short speech prompts significantly hinders the performance of
fine-grained identity imitation. In this paper, we introduce Mega-TTS 2, a
generic zero-shot multispeaker TTS model that is capable of synthesizing speech
for unseen speakers with arbitrary-length prompts. Specifically, we 1) design a
multi-reference timbre encoder to extract timbre information from multiple
reference speeches; 2) and train a prosody language model with arbitrary-length
speech prompts; With these designs, our model is suitable for prompts of
different lengths, which extends the upper bound of speech quality for
zero-shot text-to-speech. Besides arbitrary-length prompts, we introduce
arbitrary-source prompts, which leverages the probabilities derived from
multiple P-LLM outputs to produce expressive and controlled prosody.
Furthermore, we propose a phoneme-level auto-regressive duration model to
introduce in-context learning capabilities to duration modeling. Experiments
demonstrate that our method could not only synthesize identity-preserving
speech with a short prompt of an unseen speaker but also achieve improved
performance with longer speech prompts. Audio samples can be found in
https://mega-tts.github.io/mega2_demo/.
Mega-TTS 2 extends zero-shot text-to-speech with arbitrary-length prompts by incorporating a multi-reference timbre encoder and a prosody language model, enhancing both identity preservation and prosodic control.
zero-shot text-to-speech Mega-TTS 2 multi-reference timbre encoder prosody language model arbitrary-length prompts arbitrary-source prompts P-LLM phoneme-level auto-regressive duration model duration modeling