title: CapSpeech: Enabling Downstream Applications in Style-Captioned
    Text-to-Speech
summary: Recent advancements in generative artificial intelligence have significantly
  transformed the field of style-captioned text-to-speech synthesis (CapTTS).
  However, adapting CapTTS to real-world applications remains challenging due to
  the lack of standardized, comprehensive datasets and limited research on
  downstream tasks built upon CapTTS. To address these gaps, we introduce
  CapSpeech, a new benchmark designed for a series of CapTTS-related tasks,
  including style-captioned text-to-speech synthesis with sound events
  (CapTTS-SE), accent-captioned TTS (AccCapTTS), emotion-captioned TTS
  (EmoCapTTS), and text-to-speech synthesis for chat agent (AgentTTS). CapSpeech
  comprises over 10 million machine-annotated audio-caption pairs and nearly 0.36
  million human-annotated audio-caption pairs. In addition, we introduce two new
  datasets collected and recorded by a professional voice actor and experienced
  audio engineers, specifically for the AgentTTS and CapTTS-SE tasks. Alongside
  the datasets, we conduct comprehensive experiments using both autoregressive
  and non-autoregressive models on CapSpeech. Our results demonstrate
  high-fidelity and highly intelligible speech synthesis across a diverse range
  of speaking styles. To the best of our knowledge, CapSpeech is the largest
  available dataset offering comprehensive annotations for CapTTS-related tasks.
  The experiments and findings further provide valuable insights into the
  challenges of developing CapTTS systems.
paper_authors._id: 683fa315cd4314c4b3d53451
paper_authors.name: Helin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-06T07:42:05.814Z
paper_authors._id: 63ecfb5ec5b3c734085db9ed
paper_authors.avatarUrl: /avatars/0b1d03dcd7997ad1daa764fb76f88993.svg
paper_authors.fullname: Helin Wang
paper_authors.type: user
paper_authors.user: westbrook
paper_authors._id: 683fa315cd4314c4b3d53452
paper_authors.name: Jiarui Hai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-06T07:42:08.194Z
paper_authors._id: 6363e9e4969bdae89e10b5ac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6363e9e4969bdae89e10b5ac/uR0jz8pBJIQPmKC90OJgI.jpeg
paper_authors.fullname: Jiarui Hai
paper_authors.type: user
paper_authors.user: Higobeatz
paper_authors._id: 683fa315cd4314c4b3d53453
paper_authors.name: Dading Chong
paper_authors._id: 683fa315cd4314c4b3d53454
paper_authors.name: Karan Thakkar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:49:42.588Z
paper_authors._id: 634452e00a284e1e9f1d3d9d
paper_authors.avatarUrl: /avatars/127061e3785c8966fcce500a076de014.svg
paper_authors.fullname: Karan Thakkar
paper_authors.type: user
paper_authors.user: carankt
paper_authors._id: 683fa315cd4314c4b3d53455
paper_authors.name: Tiantian Feng
paper_authors._id: 683fa315cd4314c4b3d53456
paper_authors.name: Dongchao Yang
paper_authors._id: 683fa315cd4314c4b3d53457
paper_authors.name: Junhyeok Lee
paper_authors._id: 683fa315cd4314c4b3d53458
paper_authors.name: Laureano Moro Velazquez
paper_authors._id: 683fa315cd4314c4b3d53459
paper_authors.name: Jesus Villalba
paper_authors._id: 683fa315cd4314c4b3d5345a
paper_authors.name: Zengyi Qin
paper_authors._id: 683fa315cd4314c4b3d5345b
paper_authors.name: Shrikanth Narayanan
paper_authors._id: 683fa315cd4314c4b3d5345c
paper_authors.name: Mounya Elhiali
paper_authors._id: 683fa315cd4314c4b3d5345d
paper_authors.name: Najim Dehak
paper_title: CapSpeech: Enabling Downstream Applications in Style-Captioned
    Text-to-Speech
paper_summary: Recent advancements in generative artificial intelligence have significantly
  transformed the field of style-captioned text-to-speech synthesis (CapTTS).
  However, adapting CapTTS to real-world applications remains challenging due to
  the lack of standardized, comprehensive datasets and limited research on
  downstream tasks built upon CapTTS. To address these gaps, we introduce
  CapSpeech, a new benchmark designed for a series of CapTTS-related tasks,
  including style-captioned text-to-speech synthesis with sound events
  (CapTTS-SE), accent-captioned TTS (AccCapTTS), emotion-captioned TTS
  (EmoCapTTS), and text-to-speech synthesis for chat agent (AgentTTS). CapSpeech
  comprises over 10 million machine-annotated audio-caption pairs and nearly 0.36
  million human-annotated audio-caption pairs. In addition, we introduce two new
  datasets collected and recorded by a professional voice actor and experienced
  audio engineers, specifically for the AgentTTS and CapTTS-SE tasks. Alongside
  the datasets, we conduct comprehensive experiments using both autoregressive
  and non-autoregressive models on CapSpeech. Our results demonstrate
  high-fidelity and highly intelligible speech synthesis across a diverse range
  of speaking styles. To the best of our knowledge, CapSpeech is the largest
  available dataset offering comprehensive annotations for CapTTS-related tasks.
  The experiments and findings further provide valuable insights into the
  challenges of developing CapTTS systems.
paper_ai_summary: CapSpeech introduces a large benchmark dataset for various captioned text-to-speech tasks, facilitating advancements in style, accent, emotion, and chat-agent synthesis.
paper_ai_keywords: autoregressive models non-autoregressive models CapTTS CapTTS-SE AccCapTTS EmoCapTTS AgentTTS machine-annotated human-annotated