title: PromptTTS 2: Describing and Generating Voices with Text Prompt
summary: Speech conveys more information than just text, as the same word can be
  uttered in various voices to convey diverse information. Compared to
  traditional text-to-speech (TTS) methods relying on speech prompts (reference
  speech) for voice variability, using text prompts (descriptions) is more
  user-friendly since speech prompts can be hard to find or may not exist at all.
  TTS approaches based on the text prompt face two challenges: 1) the one-to-many
  problem, where not all details about voice variability can be described in the
  text prompt, and 2) the limited availability of text prompt datasets, where
  vendors and large cost of data labeling are required to write text prompt for
  speech. In this work, we introduce PromptTTS 2 to address these challenges with
  a variation network to provide variability information of voice not captured by
  text prompts, and a prompt generation pipeline to utilize the large language
  models (LLM) to compose high quality text prompts. Specifically, the variation
  network predicts the representation extracted from the reference speech (which
  contains full information about voice) based on the text prompt representation.
  For the prompt generation pipeline, it generates text prompts for speech with a
  speech understanding model to recognize voice attributes (e.g., gender, speed)
  from speech and a large language model to formulate text prompt based on the
  recognition results. Experiments on a large-scale (44K hours) speech dataset
  demonstrate that compared to the previous works, PromptTTS 2 generates voices
  more consistent with text prompts and supports the sampling of diverse voice
  variability, thereby offering users more choices on voice generation.
  Additionally, the prompt generation pipeline produces high-quality prompts,
  eliminating the large labeling cost. The demo page of PromptTTS 2 is available
  onlinehttps://speechresearch.github.io/prompttts2.
paper_authors._id: 64f7edc60590f3db148a17b1
paper_authors.name: Yichong Leng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-06T11:16:40.687Z
paper_authors._id: 64a0347b528a9bbe59d6e08c
paper_authors.avatarUrl: /avatars/6dd0bad84d711d1048a0a4169e621773.svg
paper_authors.fullname: Yichong Leng
paper_authors.type: user
paper_authors.user: ustcscallion
paper_authors._id: 64f7edc60590f3db148a17b2
paper_authors.name: Zhifang Guo
paper_authors._id: 64f7edc60590f3db148a17b3
paper_authors.name: Kai Shen
paper_authors._id: 64f7edc60590f3db148a17b4
paper_authors.name: Xu Tan
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-05T05:51:01.425Z
paper_authors._id: 5f1040b6e9d71719e3be71d2
paper_authors.avatarUrl: /avatars/a2f28940236ae625ed3810ad62e343ff.svg
paper_authors.fullname: Xu Tan
paper_authors.type: user
paper_authors.user: xutan
paper_authors._id: 64f7edc60590f3db148a17b5
paper_authors.name: Zeqian Ju
paper_authors._id: 64f7edc60590f3db148a17b6
paper_authors.name: Yanqing Liu
paper_authors._id: 64f7edc60590f3db148a17b7
paper_authors.name: Yufei Liu
paper_authors._id: 64f7edc60590f3db148a17b8
paper_authors.name: Dongchao Yang
paper_authors._id: 64f7edc60590f3db148a17b9
paper_authors.name: Leying Zhang
paper_authors._id: 64f7edc60590f3db148a17ba
paper_authors.name: Kaitao Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-06T11:19:10.945Z
paper_authors._id: 5fc0b2b61160c47d1d438568
paper_authors.avatarUrl: /avatars/90beea6b452c662d579197dbf592423a.svg
paper_authors.fullname: Kaitao Song
paper_authors.type: user
paper_authors.user: KaitaoSong
paper_authors._id: 64f7edc60590f3db148a17bb
paper_authors.name: Lei He
paper_authors._id: 64f7edc60590f3db148a17bc
paper_authors.name: Xiang-Yang Li
paper_authors._id: 64f7edc60590f3db148a17bd
paper_authors.name: Sheng Zhao
paper_authors._id: 64f7edc60590f3db148a17be
paper_authors.name: Tao Qin
paper_authors._id: 64f7edc60590f3db148a17bf
paper_authors.name: Jiang Bian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-06T11:22:32.148Z
paper_authors._id: 63f253f8f4e30ffd2bd308fb
paper_authors.avatarUrl: /avatars/303f4c7ee588f638acf78a7966786e1e.svg
paper_authors.fullname: Jiang Bian
paper_authors.type: user
paper_authors.user: bianjiang
paper_title: PromptTTS 2: Describing and Generating Voices with Text Prompt
paper_summary: Speech conveys more information than just text, as the same word can be
  uttered in various voices to convey diverse information. Compared to
  traditional text-to-speech (TTS) methods relying on speech prompts (reference
  speech) for voice variability, using text prompts (descriptions) is more
  user-friendly since speech prompts can be hard to find or may not exist at all.
  TTS approaches based on the text prompt face two challenges: 1) the one-to-many
  problem, where not all details about voice variability can be described in the
  text prompt, and 2) the limited availability of text prompt datasets, where
  vendors and large cost of data labeling are required to write text prompt for
  speech. In this work, we introduce PromptTTS 2 to address these challenges with
  a variation network to provide variability information of voice not captured by
  text prompts, and a prompt generation pipeline to utilize the large language
  models (LLM) to compose high quality text prompts. Specifically, the variation
  network predicts the representation extracted from the reference speech (which
  contains full information about voice) based on the text prompt representation.
  For the prompt generation pipeline, it generates text prompts for speech with a
  speech understanding model to recognize voice attributes (e.g., gender, speed)
  from speech and a large language model to formulate text prompt based on the
  recognition results. Experiments on a large-scale (44K hours) speech dataset
  demonstrate that compared to the previous works, PromptTTS 2 generates voices
  more consistent with text prompts and supports the sampling of diverse voice
  variability, thereby offering users more choices on voice generation.
  Additionally, the prompt generation pipeline produces high-quality prompts,
  eliminating the large labeling cost. The demo page of PromptTTS 2 is available
  onlinehttps://speechresearch.github.io/prompttts2.
paper_ai_keywords: PromptTTS 2 variation network prompt generation pipeline large language models (LLM) text prompt representation speech understanding model voice attributes gender speed voice variability reference speech