title: CLAPSpeech: Learning Prosody from Text Context with Contrastive
    Language-Audio Pre-training
summary: Improving text representation has attracted much attention to achieve
  expressive text-to-speech (TTS). However, existing works only implicitly learn
  the prosody with masked token reconstruction tasks, which leads to low training
  efficiency and difficulty in prosody modeling. We propose CLAPSpeech, a
  cross-modal contrastive pre-training framework that explicitly learns the
  prosody variance of the same text token under different contexts. Specifically,
  1) We encourage the model to connect the text context with its corresponding
  prosody pattern in the joint multi-modal space with the elaborate design of the
  encoder inputs and contrastive loss; 2) We introduce a multi-scale pre-training
  pipeline to capture prosody patterns in multiple levels. We show how to
  incorporate CLAPSpeech into existing TTS models for better prosody. Experiments
  on three datasets not only show that CLAPSpeech could improve the prosody
  prediction for existing TTS methods, but also demonstrate its generalization
  ability to adapt to multiple languages and multi-speaker TTS. We also deeply
  analyze the principle behind the performance of CLAPSpeech. Ablation studies
  demonstrate the necessity of each component in our method. Source code and
  audio samples are available at https://clapspeech.github.io.
paper_authors._id: 646700eff3e78d1d6be121a3
paper_authors.name: Zhenhui Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-21T20:41:10.207Z
paper_authors._id: 6277fcd8c5c547c31b80cc77
paper_authors.avatarUrl: /avatars/7dab3f5c879b5c5eb86f822119a9b5bb.svg
paper_authors.fullname: Zhenhui Ye
paper_authors.type: user
paper_authors.user: yerfor
paper_authors._id: 646700eff3e78d1d6be121a4
paper_authors.name: Rongjie Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-21T20:41:54.058Z
paper_authors._id: 62f6556e04e5e02f82ac8531
paper_authors.avatarUrl: /avatars/01024815aeaaf8c4721e937b3db68903.svg
paper_authors.fullname: Rongjiehuang
paper_authors.type: user
paper_authors.user: Rongjiehuang
paper_authors._id: 646700eff3e78d1d6be121a5
paper_authors.name: Yi Ren
paper_authors._id: 646700eff3e78d1d6be121a6
paper_authors.name: Ziyue Jiang
paper_authors._id: 646700eff3e78d1d6be121a7
paper_authors.name: Jinglin Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-21T20:42:36.814Z
paper_authors._id: 6205c6ad00e630c7a5d0fc2e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1645184905991-6205c6ad00e630c7a5d0fc2e.jpeg
paper_authors.fullname: Jinglin Liu
paper_authors.type: user
paper_authors.user: Silentlin
paper_authors._id: 646700eff3e78d1d6be121a8
paper_authors.name: Jinzheng He
paper_authors._id: 646700eff3e78d1d6be121a9
paper_authors.name: Xiang Yin
paper_authors._id: 646700eff3e78d1d6be121aa
paper_authors.name: Zhou Zhao
paper_title: CLAPSpeech: Learning Prosody from Text Context with Contrastive
    Language-Audio Pre-training
paper_summary: Improving text representation has attracted much attention to achieve
  expressive text-to-speech (TTS). However, existing works only implicitly learn
  the prosody with masked token reconstruction tasks, which leads to low training
  efficiency and difficulty in prosody modeling. We propose CLAPSpeech, a
  cross-modal contrastive pre-training framework that explicitly learns the
  prosody variance of the same text token under different contexts. Specifically,
  1) We encourage the model to connect the text context with its corresponding
  prosody pattern in the joint multi-modal space with the elaborate design of the
  encoder inputs and contrastive loss; 2) We introduce a multi-scale pre-training
  pipeline to capture prosody patterns in multiple levels. We show how to
  incorporate CLAPSpeech into existing TTS models for better prosody. Experiments
  on three datasets not only show that CLAPSpeech could improve the prosody
  prediction for existing TTS methods, but also demonstrate its generalization
  ability to adapt to multiple languages and multi-speaker TTS. We also deeply
  analyze the principle behind the performance of CLAPSpeech. Ablation studies
  demonstrate the necessity of each component in our method. Source code and
  audio samples are available at https://clapspeech.github.io.
paper_ai_keywords: cross-modal contrastive pre-training prosody variance textual context prosody pattern joint multi-modal space contrastive loss Multi-scale pre-training prosody prediction generalization ability multi-speaker TTS