title: VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text
    to Speech Synthesizers
summary: This paper introduces VALL-E 2, the latest advancement in neural codec
  language models that marks a milestone in zero-shot text-to-speech synthesis
  (TTS), achieving human parity for the first time. Based on its predecessor,
  VALL-E, the new iteration introduces two significant enhancements: Repetition
  Aware Sampling refines the original nucleus sampling process by accounting for
  token repetition in the decoding history. It not only stabilizes the decoding
  but also circumvents the infinite loop issue. Grouped Code Modeling organizes
  codec codes into groups to effectively shorten the sequence length, which not
  only boosts inference speed but also addresses the challenges of long sequence
  modeling. Our experiments on the LibriSpeech and VCTK datasets show that VALL-E
  2 surpasses previous systems in speech robustness, naturalness, and speaker
  similarity. It is the first of its kind to reach human parity on these
  benchmarks. Moreover, VALL-E 2 consistently synthesizes high-quality speech,
  even for sentences that are traditionally challenging due to their complexity
  or repetitive phrases. The advantages of this work could contribute to valuable
  endeavors, such as generating speech for individuals with aphasia or people
  with amyotrophic lateral sclerosis. Demos of VALL-E 2 will be posted to
  https://aka.ms/valle2.
paper_authors._id: 6667c3d28b7703d4b30a5d32
paper_authors.name: Sanyuan Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:16:32.142Z
paper_authors._id: 6465843ce27766e8921b9cab
paper_authors.avatarUrl: /avatars/6017db24da593c9afb21edfcedcddfd4.svg
paper_authors.fullname: Sanyuan Chen
paper_authors.type: user
paper_authors.user: Sanyuan-Chen
paper_authors._id: 6667c3d28b7703d4b30a5d33
paper_authors.name: Shujie Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:16:38.358Z
paper_authors._id: 64dc94f1bdac91d86834e13b
paper_authors.avatarUrl: /avatars/264ebe3867121f56eefac6e1ab8cf17f.svg
paper_authors.fullname: Shujie Liu
paper_authors.type: user
paper_authors.user: j4ckl1u
paper_authors._id: 6667c3d28b7703d4b30a5d34
paper_authors.name: Long Zhou
paper_authors._id: 6667c3d28b7703d4b30a5d35
paper_authors.name: Yanqing Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:26:54.652Z
paper_authors._id: 654a771a2d2fcd6bf27ff0de
paper_authors.avatarUrl: /avatars/6312b23c642009f13c7993ce4658d06d.svg
paper_authors.fullname: Liuyanqing
paper_authors.type: user
paper_authors.user: Yanqing0327
paper_authors._id: 6667c3d28b7703d4b30a5d36
paper_authors.name: Xu Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:27:07.844Z
paper_authors._id: 5f1040b6e9d71719e3be71d2
paper_authors.avatarUrl: /avatars/a2f28940236ae625ed3810ad62e343ff.svg
paper_authors.fullname: Xu Tan
paper_authors.type: user
paper_authors.user: xutan
paper_authors._id: 6667c3d28b7703d4b30a5d37
paper_authors.name: Jinyu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:27:30.230Z
paper_authors._id: 644dffbe16703fd6702d3bd5
paper_authors.avatarUrl: /avatars/dab5831f689ffec9cd1f63213539d3f0.svg
paper_authors.fullname: Jinyu Li
paper_authors.type: user
paper_authors.user: JinyuID
paper_authors._id: 6667c3d28b7703d4b30a5d38
paper_authors.name: Sheng Zhao
paper_authors._id: 6667c3d28b7703d4b30a5d39
paper_authors.name: Yao Qian
paper_authors._id: 6667c3d28b7703d4b30a5d3a
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:25:59.688Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text
    to Speech Synthesizers
paper_summary: This paper introduces VALL-E 2, the latest advancement in neural codec
  language models that marks a milestone in zero-shot text-to-speech synthesis
  (TTS), achieving human parity for the first time. Based on its predecessor,
  VALL-E, the new iteration introduces two significant enhancements: Repetition
  Aware Sampling refines the original nucleus sampling process by accounting for
  token repetition in the decoding history. It not only stabilizes the decoding
  but also circumvents the infinite loop issue. Grouped Code Modeling organizes
  codec codes into groups to effectively shorten the sequence length, which not
  only boosts inference speed but also addresses the challenges of long sequence
  modeling. Our experiments on the LibriSpeech and VCTK datasets show that VALL-E
  2 surpasses previous systems in speech robustness, naturalness, and speaker
  similarity. It is the first of its kind to reach human parity on these
  benchmarks. Moreover, VALL-E 2 consistently synthesizes high-quality speech,
  even for sentences that are traditionally challenging due to their complexity
  or repetitive phrases. The advantages of this work could contribute to valuable
  endeavors, such as generating speech for individuals with aphasia or people
  with amyotrophic lateral sclerosis. Demos of VALL-E 2 will be posted to
  https://aka.ms/valle2.
paper_ai_keywords: neural codec language models zero-shot text-to-speech synthesis (TTS) human parity Repetition Aware Sampling nucleus sampling decoding history infinite loop issue Grouped Code Modeling sequence length inference speed long sequence modeling LibriSpeech VCTK speech robustness naturalness speaker similarity high-quality speech aphasia amyotrophic lateral sclerosis