title: FlashSpeech: Efficient Zero-Shot Speech Synthesis
summary: Recent progress in large-scale zero-shot speech synthesis has been
  significantly advanced by language models and diffusion models. However, the
  generation process of both methods is slow and computationally intensive.
  Efficient speech synthesis using a lower computing budget to achieve quality on
  par with previous work remains a significant challenge. In this paper, we
  present FlashSpeech, a large-scale zero-shot speech synthesis system with
  approximately 5\% of the inference time compared with previous work.
  FlashSpeech is built on the latent consistency model and applies a novel
  adversarial consistency training approach that can train from scratch without
  the need for a pre-trained diffusion model as the teacher. Furthermore, a new
  prosody generator module enhances the diversity of prosody, making the rhythm
  of the speech sound more natural. The generation processes of FlashSpeech can
  be achieved efficiently with one or two sampling steps while maintaining high
  audio quality and high similarity to the audio prompt for zero-shot speech
  generation. Our experimental results demonstrate the superior performance of
  FlashSpeech. Notably, FlashSpeech can be about 20 times faster than other
  zero-shot speech synthesis systems while maintaining comparable performance in
  terms of voice quality and similarity. Furthermore, FlashSpeech demonstrates
  its versatility by efficiently performing tasks like voice conversion, speech
  editing, and diverse speech sampling. Audio samples can be found in
  https://flashspeech.github.io/.
paper_authors._id: 662868cc189fe0763bb8a2fe
paper_authors.name: Zhen Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:58:36.245Z
paper_authors._id: 645f172d7c6bff8577353d1a
paper_authors.avatarUrl: /avatars/a83682e1343809257b082b78d58c582a.svg
paper_authors.fullname: ZhenYE
paper_authors.type: user
paper_authors.user: ZhenYe234
paper_authors._id: 662868cc189fe0763bb8a2ff
paper_authors.name: Zeqian Ju
paper_authors._id: 662868cc189fe0763bb8a300
paper_authors.name: Haohe Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-24T12:05:52.736Z
paper_authors._id: 6155245d1c762d4d61b51d5d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677233708065-6155245d1c762d4d61b51d5d.png
paper_authors.fullname: haoheliu
paper_authors.type: user
paper_authors.user: haoheliu
paper_authors._id: 662868cc189fe0763bb8a301
paper_authors.name: Xu Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-24T12:06:12.115Z
paper_authors._id: 5f1040b6e9d71719e3be71d2
paper_authors.avatarUrl: /avatars/a2f28940236ae625ed3810ad62e343ff.svg
paper_authors.fullname: Xu Tan
paper_authors.type: user
paper_authors.user: xutan
paper_authors._id: 662868cc189fe0763bb8a302
paper_authors.name: Jianyi Chen
paper_authors._id: 662868cc189fe0763bb8a303
paper_authors.name: Yiwen Lu
paper_authors._id: 662868cc189fe0763bb8a304
paper_authors.name: Peiwen Sun
paper_authors._id: 662868cc189fe0763bb8a305
paper_authors.name: Jiahao Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-13T15:40:04.783Z
paper_authors._id: 647f7cf4b0e9676458a014f9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647f7cf4b0e9676458a014f9/4L_0MbVErSmPYrk3mIzNP.jpeg
paper_authors.fullname: jiahao
paper_authors.type: user
paper_authors.user: cherishpjh
paper_authors._id: 662868cc189fe0763bb8a306
paper_authors.name: Weizhen Bian
paper_authors._id: 662868cc189fe0763bb8a307
paper_authors.name: Shulin He
paper_authors._id: 662868cc189fe0763bb8a308
paper_authors.name: Qifeng Liu
paper_authors._id: 662868cc189fe0763bb8a309
paper_authors.name: Yike Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-24T12:07:49.056Z
paper_authors._id: 64762b325f70f9b2d0ade28e
paper_authors.avatarUrl: /avatars/1f37382724b475ece805f943a8858acd.svg
paper_authors.fullname: Yike Guo
paper_authors.type: user
paper_authors.user: SuaLily
paper_authors._id: 662868cc189fe0763bb8a30a
paper_authors.name: Wei Xue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-24T11:11:21.136Z
paper_authors._id: 6628adb14277eae0da5eee28
paper_authors.avatarUrl: /avatars/6cb41b80cc5e014e455dfc2a22682e64.svg
paper_authors.fullname: HKUST Audio
paper_authors.type: user
paper_authors.user: HKUST-Audio
paper_title: FlashSpeech: Efficient Zero-Shot Speech Synthesis
paper_summary: Recent progress in large-scale zero-shot speech synthesis has been
  significantly advanced by language models and diffusion models. However, the
  generation process of both methods is slow and computationally intensive.
  Efficient speech synthesis using a lower computing budget to achieve quality on
  par with previous work remains a significant challenge. In this paper, we
  present FlashSpeech, a large-scale zero-shot speech synthesis system with
  approximately 5\% of the inference time compared with previous work.
  FlashSpeech is built on the latent consistency model and applies a novel
  adversarial consistency training approach that can train from scratch without
  the need for a pre-trained diffusion model as the teacher. Furthermore, a new
  prosody generator module enhances the diversity of prosody, making the rhythm
  of the speech sound more natural. The generation processes of FlashSpeech can
  be achieved efficiently with one or two sampling steps while maintaining high
  audio quality and high similarity to the audio prompt for zero-shot speech
  generation. Our experimental results demonstrate the superior performance of
  FlashSpeech. Notably, FlashSpeech can be about 20 times faster than other
  zero-shot speech synthesis systems while maintaining comparable performance in
  terms of voice quality and similarity. Furthermore, FlashSpeech demonstrates
  its versatility by efficiently performing tasks like voice conversion, speech
  editing, and diverse speech sampling. Audio samples can be found in
  https://flashspeech.github.io/.
paper_ai_keywords: zero-shot speech synthesis language models diffusion models alpha matte inference time latent consistency model adversarial consistency training approach prosody generator sampling steps audio quality voice quality similarity voice conversion speech editing diverse speech sampling