PromptTTS 2: Describing and Generating Voices with Text Prompt
Speech conveys more information than just text, as the same word can be
uttered in various voices to convey diverse information. Compared to
traditional text-to-speech (TTS) methods relying on speech prompts (reference
speech) for voice variability, using text prompts (descriptions) is more
user-friendly since speech prompts can be hard to find or may not exist at all.
TTS approaches based on the text prompt face two challenges: 1) the one-to-many
problem, where not all details about voice variability can be described in the
text prompt, and 2) the limited availability of text prompt datasets, where
vendors and large cost of data labeling are required to write text prompt for
speech. In this work, we introduce PromptTTS 2 to address these challenges with
a variation network to provide variability information of voice not captured by
text prompts, and a prompt generation pipeline to utilize the large language
models (LLM) to compose high quality text prompts. Specifically, the variation
network predicts the representation extracted from the reference speech (which
contains full information about voice) based on the text prompt representation.
For the prompt generation pipeline, it generates text prompts for speech with a
speech understanding model to recognize voice attributes (e.g., gender, speed)
from speech and a large language model to formulate text prompt based on the
recognition results. Experiments on a large-scale (44K hours) speech dataset
demonstrate that compared to the previous works, PromptTTS 2 generates voices
more consistent with text prompts and supports the sampling of diverse voice
variability, thereby offering users more choices on voice generation.
Additionally, the prompt generation pipeline produces high-quality prompts,
eliminating the large labeling cost. The demo page of PromptTTS 2 is available
onlinehttps://speechresearch.github.io/prompttts2.
64f7edc60590f3db148a17b1
Yichong Leng
admin_assigned
2023-09-06T11:16:40.687Z
64a0347b528a9bbe59d6e08c
/avatars/6dd0bad84d711d1048a0a4169e621773.svg
Yichong Leng
user
ustcscallion
64f7edc60590f3db148a17b2
Zhifang Guo
64f7edc60590f3db148a17b3
Kai Shen
64f7edc60590f3db148a17b4
Xu Tan
extracted_confirmed
2024-04-05T05:51:01.425Z
5f1040b6e9d71719e3be71d2
/avatars/a2f28940236ae625ed3810ad62e343ff.svg
Xu Tan
user
xutan
64f7edc60590f3db148a17b5
Zeqian Ju
64f7edc60590f3db148a17b6
Yanqing Liu
64f7edc60590f3db148a17b7
Yufei Liu
64f7edc60590f3db148a17b8
Dongchao Yang
64f7edc60590f3db148a17b9
Leying Zhang
64f7edc60590f3db148a17ba
Kaitao Song
admin_assigned
2023-09-06T11:19:10.945Z
5fc0b2b61160c47d1d438568
/avatars/b355912b0ec683e73f21c8d36620e146.svg
Kaitao Song
user
KaitaoSong
64f7edc60590f3db148a17bb
Lei He
64f7edc60590f3db148a17bc
Xiang-Yang Li
64f7edc60590f3db148a17bd
Sheng Zhao
64f7edc60590f3db148a17be
Tao Qin
64f7edc60590f3db148a17bf
Jiang Bian
admin_assigned
2023-09-06T11:22:32.148Z
63f253f8f4e30ffd2bd308fb
/avatars/303f4c7ee588f638acf78a7966786e1e.svg
Jiang Bian
user
bianjiang
PromptTTS 2: Describing and Generating Voices with Text Prompt
Speech conveys more information than just text, as the same word can be
uttered in various voices to convey diverse information. Compared to
traditional text-to-speech (TTS) methods relying on speech prompts (reference
speech) for voice variability, using text prompts (descriptions) is more
user-friendly since speech prompts can be hard to find or may not exist at all.
TTS approaches based on the text prompt face two challenges: 1) the one-to-many
problem, where not all details about voice variability can be described in the
text prompt, and 2) the limited availability of text prompt datasets, where
vendors and large cost of data labeling are required to write text prompt for
speech. In this work, we introduce PromptTTS 2 to address these challenges with
a variation network to provide variability information of voice not captured by
text prompts, and a prompt generation pipeline to utilize the large language
models (LLM) to compose high quality text prompts. Specifically, the variation
network predicts the representation extracted from the reference speech (which
contains full information about voice) based on the text prompt representation.
For the prompt generation pipeline, it generates text prompts for speech with a
speech understanding model to recognize voice attributes (e.g., gender, speed)
from speech and a large language model to formulate text prompt based on the
recognition results. Experiments on a large-scale (44K hours) speech dataset
demonstrate that compared to the previous works, PromptTTS 2 generates voices
more consistent with text prompts and supports the sampling of diverse voice
variability, thereby offering users more choices on voice generation.
Additionally, the prompt generation pipeline produces high-quality prompts,
eliminating the large labeling cost. The demo page of PromptTTS 2 is available
onlinehttps://speechresearch.github.io/prompttts2.
PromptTTS 2 addresses the challenges in text-to-speech systems by using a variation network to capture voice variability not expressed in text prompts and a prompt generation pipeline utilizing large language models to create high-quality text prompts.
text-to-speech text prompts voice variability variation network large language models prompt generation pipeline reference speech speech understanding model speech attributes diverse voice variability