title: Pheme: Efficient and Conversational Speech Generation
summary: In recent years, speech generation has seen remarkable progress, now
  achieving one-shot generation capability that is often virtually
  indistinguishable from real human voice. Integrating such advancements in
  speech generation with large language models might revolutionize a wide range
  of applications. However, certain applications, such as assistive
  conversational systems, require natural and conversational speech generation
  tools that also operate efficiently in real time. Current state-of-the-art
  models like VALL-E and SoundStorm, powered by hierarchical neural audio codecs,
  require large neural components and extensive training data to work well. In
  contrast, MQTTS aims to build more compact conversational TTS models while
  capitalizing on smaller-scale real-life conversational speech data. However,
  its autoregressive nature yields high inference latency and thus limits its
  real-time usage. In order to mitigate the current limitations of the
  state-of-the-art TTS models while capitalizing on their strengths, in this work
  we introduce the Pheme model series that 1) offers compact yet high-performing
  models, 2) allows for parallel speech generation of 3) natural conversational
  speech, and 4) it can be trained efficiently on smaller-scale conversational
  data, cutting data demands by more than 10x but still matching the quality of
  the autoregressive TTS models. We also show that through simple teacher-student
  distillation we can meet significant improvements in voice quality for
  single-speaker setups on top of pretrained Pheme checkpoints, relying solely on
  synthetic speech generated by much larger teacher models. Audio samples and
  pretrained models are available online.
paper_authors._id: 659b6086eff07dcf1fede121
paper_authors.name: Paweł Budzianowski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:43:53.695Z
paper_authors._id: 6273c47ef6d63a28483eaba2
paper_authors.avatarUrl: /avatars/f50e8c37f070846814985091609aa23f.svg
paper_authors.fullname: Paweł Budzianowski
paper_authors.type: user
paper_authors.user: pfb30
paper_authors._id: 659b6086eff07dcf1fede122
paper_authors.name: Taras Sereda
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:44:16.608Z
paper_authors._id: 636a9312d5ee5ba5bbb32a1e
paper_authors.avatarUrl: /avatars/58ce173bcac7432cfc9fbac9712ff844.svg
paper_authors.fullname: Taras Sereda
paper_authors.type: user
paper_authors.user: taras-sereda
paper_authors._id: 659b6086eff07dcf1fede123
paper_authors.name: Tomasz Cichy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:44:35.504Z
paper_authors._id: 652eb092338c761caf2764ba
paper_authors.avatarUrl: /avatars/7ab894033e2f3bffc8f793aee32eb510.svg
paper_authors.fullname: Tomasz Cichy
paper_authors.type: user
paper_authors.user: cichyt
paper_authors._id: 659b6086eff07dcf1fede124
paper_authors.name: Ivan Vulić
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:44:42.167Z
paper_authors._id: 6273e70dc8d55dd434bd8e52
paper_authors.avatarUrl: /avatars/3483eeda218e95b1eb00c3dc63c7d000.svg
paper_authors.fullname: Ivan Vulić
paper_authors.type: user
paper_authors.user: ivulic
paper_title: Pheme: Efficient and Conversational Speech Generation
paper_summary: In recent years, speech generation has seen remarkable progress, now
  achieving one-shot generation capability that is often virtually
  indistinguishable from real human voice. Integrating such advancements in
  speech generation with large language models might revolutionize a wide range
  of applications. However, certain applications, such as assistive
  conversational systems, require natural and conversational speech generation
  tools that also operate efficiently in real time. Current state-of-the-art
  models like VALL-E and SoundStorm, powered by hierarchical neural audio codecs,
  require large neural components and extensive training data to work well. In
  contrast, MQTTS aims to build more compact conversational TTS models while
  capitalizing on smaller-scale real-life conversational speech data. However,
  its autoregressive nature yields high inference latency and thus limits its
  real-time usage. In order to mitigate the current limitations of the
  state-of-the-art TTS models while capitalizing on their strengths, in this work
  we introduce the Pheme model series that 1) offers compact yet high-performing
  models, 2) allows for parallel speech generation of 3) natural conversational
  speech, and 4) it can be trained efficiently on smaller-scale conversational
  data, cutting data demands by more than 10x but still matching the quality of
  the autoregressive TTS models. We also show that through simple teacher-student
  distillation we can meet significant improvements in voice quality for
  single-speaker setups on top of pretrained Pheme checkpoints, relying solely on
  synthetic speech generated by much larger teacher models. Audio samples and
  pretrained models are available online.
paper_ai_summary: The Pheme model series achieves compact and high-quality voice generation with parallel processing, efficient training on smaller datasets, and improved voice quality through distillation.
paper_ai_keywords: speech generation hierarchical neural audio codecs TTS models autoregressive nature real-time usage compact models parallel speech generation natural conversational speech teacher-student distillation pretrained Pheme checkpoints