title: HierSpeech++: Bridging the Gap between Semantic and Acoustic
    Representation of Speech by Hierarchical Variational Inference for Zero-shot
    Speech Synthesis
summary: Large language models (LLM)-based speech synthesis has been widely adopted in
  zero-shot speech synthesis. However, they require a large-scale data and
  possess the same limitations as previous autoregressive speech models,
  including slow inference speed and lack of robustness. This paper proposes
  HierSpeech++, a fast and strong zero-shot speech synthesizer for text-to-speech
  (TTS) and voice conversion (VC). We verified that hierarchical speech synthesis
  frameworks could significantly improve the robustness and expressiveness of the
  synthetic speech. Furthermore, we significantly improve the naturalness and
  speaker similarity of synthetic speech even in zero-shot speech synthesis
  scenarios. For text-to-speech, we adopt the text-to-vec framework, which
  generates a self-supervised speech representation and an F0 representation
  based on text representations and prosody prompts. Then, HierSpeech++ generates
  speech from the generated vector, F0, and voice prompt. We further introduce a
  high-efficient speech super-resolution framework from 16 kHz to 48 kHz. The
  experimental results demonstrated that the hierarchical variational autoencoder
  could be a strong zero-shot speech synthesizer given that it outperforms
  LLM-based and diffusion-based models. Moreover, we achieved the first
  human-level quality zero-shot speech synthesis. Audio samples and source code
  are available at https://github.com/sh-lee-prml/HierSpeechpp.
paper_authors._id: 655d90c9bf82497038e70802
paper_authors.name: Sang-Hoon Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-22T12:50:49.532Z
paper_authors._id: 654a62c211a85feec88ef8f2
paper_authors.avatarUrl: /avatars/aeff2661e7f3cbc6b3cdf43c6e31ca9b.svg
paper_authors.fullname: Sang-Hoon Lee
paper_authors.type: user
paper_authors.user: LeeSangHoon
paper_authors._id: 655d90c9bf82497038e70803
paper_authors.name: Ha-Yeong Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-22T12:50:56.373Z
paper_authors._id: 6401a0c14a5c92eccfe333c2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6401a0c14a5c92eccfe333c2/-fv3jekUxMNrE058phGpA.png
paper_authors.fullname: Ha-Yeong Choi
paper_authors.type: user
paper_authors.user: Ha0
paper_authors._id: 655d90c9bf82497038e70804
paper_authors.name: Seung-Bin Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-22T13:11:26.639Z
paper_authors._id: 655dfc0cf828e5ad0f1f833f
paper_authors.avatarUrl: /avatars/2a36974ba91aead89b29a35265fe2398.svg
paper_authors.fullname: Seung-Bin Kim
paper_authors.type: user
paper_authors.user: sb-kim
paper_authors._id: 655d90c9bf82497038e70805
paper_authors.name: Seong-Whan Lee
paper_title: HierSpeech++: Bridging the Gap between Semantic and Acoustic
    Representation of Speech by Hierarchical Variational Inference for Zero-shot
    Speech Synthesis
paper_summary: Large language models (LLM)-based speech synthesis has been widely adopted in
  zero-shot speech synthesis. However, they require a large-scale data and
  possess the same limitations as previous autoregressive speech models,
  including slow inference speed and lack of robustness. This paper proposes
  HierSpeech++, a fast and strong zero-shot speech synthesizer for text-to-speech
  (TTS) and voice conversion (VC). We verified that hierarchical speech synthesis
  frameworks could significantly improve the robustness and expressiveness of the
  synthetic speech. Furthermore, we significantly improve the naturalness and
  speaker similarity of synthetic speech even in zero-shot speech synthesis
  scenarios. For text-to-speech, we adopt the text-to-vec framework, which
  generates a self-supervised speech representation and an F0 representation
  based on text representations and prosody prompts. Then, HierSpeech++ generates
  speech from the generated vector, F0, and voice prompt. We further introduce a
  high-efficient speech super-resolution framework from 16 kHz to 48 kHz. The
  experimental results demonstrated that the hierarchical variational autoencoder
  could be a strong zero-shot speech synthesizer given that it outperforms
  LLM-based and diffusion-based models. Moreover, we achieved the first
  human-level quality zero-shot speech synthesis. Audio samples and source code
  are available at https://github.com/sh-lee-prml/HierSpeechpp.
paper_ai_keywords: zero-shot speech synthesis autoregressive speech models HierSpeech++ controlled text-to-speech (TTS) voice conversion (VC) hierarchical speech synthesis text-to-vec framework self-supervised speech representation F0 representation hierarchical variational autoencoder super-resolution framework diffusion-based models human-level quality zero-shot speech synthesis