Pheme: Efficient and Conversational Speech Generation
In recent years, speech generation has seen remarkable progress, now
achieving one-shot generation capability that is often virtually
indistinguishable from real human voice. Integrating such advancements in
speech generation with large language models might revolutionize a wide range
of applications. However, certain applications, such as assistive
conversational systems, require natural and conversational speech generation
tools that also operate efficiently in real time. Current state-of-the-art
models like VALL-E and SoundStorm, powered by hierarchical neural audio codecs,
require large neural components and extensive training data to work well. In
contrast, MQTTS aims to build more compact conversational TTS models while
capitalizing on smaller-scale real-life conversational speech data. However,
its autoregressive nature yields high inference latency and thus limits its
real-time usage. In order to mitigate the current limitations of the
state-of-the-art TTS models while capitalizing on their strengths, in this work
we introduce the Pheme model series that 1) offers compact yet high-performing
models, 2) allows for parallel speech generation of 3) natural conversational
speech, and 4) it can be trained efficiently on smaller-scale conversational
data, cutting data demands by more than 10x but still matching the quality of
the autoregressive TTS models. We also show that through simple teacher-student
distillation we can meet significant improvements in voice quality for
single-speaker setups on top of pretrained Pheme checkpoints, relying solely on
synthetic speech generated by much larger teacher models. Audio samples and
pretrained models are available online.
659b6086eff07dcf1fede121
Paweł Budzianowski
admin_assigned
2024-01-08T11:43:53.695Z
6273c47ef6d63a28483eaba2
/avatars/f50e8c37f070846814985091609aa23f.svg
Paweł Budzianowski
user
pfb30
659b6086eff07dcf1fede122
Taras Sereda
admin_assigned
2024-01-08T11:44:16.608Z
636a9312d5ee5ba5bbb32a1e
/avatars/58ce173bcac7432cfc9fbac9712ff844.svg
Taras Sereda
user
taras-sereda
659b6086eff07dcf1fede123
Tomasz Cichy
admin_assigned
2024-01-08T11:44:35.504Z
652eb092338c761caf2764ba
/avatars/7ab894033e2f3bffc8f793aee32eb510.svg
Tomasz Cichy
user
cichyt
659b6086eff07dcf1fede124
Ivan Vulić
admin_assigned
2024-01-08T11:44:42.167Z
6273e70dc8d55dd434bd8e52
/avatars/3483eeda218e95b1eb00c3dc63c7d000.svg
Ivan Vulić
user
ivulic
Pheme: Efficient and Conversational Speech Generation
In recent years, speech generation has seen remarkable progress, now
achieving one-shot generation capability that is often virtually
indistinguishable from real human voice. Integrating such advancements in
speech generation with large language models might revolutionize a wide range
of applications. However, certain applications, such as assistive
conversational systems, require natural and conversational speech generation
tools that also operate efficiently in real time. Current state-of-the-art
models like VALL-E and SoundStorm, powered by hierarchical neural audio codecs,
require large neural components and extensive training data to work well. In
contrast, MQTTS aims to build more compact conversational TTS models while
capitalizing on smaller-scale real-life conversational speech data. However,
its autoregressive nature yields high inference latency and thus limits its
real-time usage. In order to mitigate the current limitations of the
state-of-the-art TTS models while capitalizing on their strengths, in this work
we introduce the Pheme model series that 1) offers compact yet high-performing
models, 2) allows for parallel speech generation of 3) natural conversational
speech, and 4) it can be trained efficiently on smaller-scale conversational
data, cutting data demands by more than 10x but still matching the quality of
the autoregressive TTS models. We also show that through simple teacher-student
distillation we can meet significant improvements in voice quality for
single-speaker setups on top of pretrained Pheme checkpoints, relying solely on
synthetic speech generated by much larger teacher models. Audio samples and
pretrained models are available online.
The Pheme model series achieves compact and high-quality voice generation with parallel processing, efficient training on smaller datasets, and improved voice quality through distillation.
speech generation hierarchical neural audio codecs TTS models autoregressive nature real-time usage compact models parallel speech generation natural conversational speech teacher-student distillation pretrained Pheme checkpoints