title: NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and
    Diffusion Models
summary: While recent large-scale text-to-speech (TTS) models have achieved
  significant progress, they still fall short in speech quality, similarity, and
  prosody. Considering speech intricately encompasses various attributes (e.g.,
  content, prosody, timbre, and acoustic details) that pose significant
  challenges for generation, a natural idea is to factorize speech into
  individual subspaces representing different attributes and generate them
  individually. Motivated by it, we propose NaturalSpeech 3, a TTS system with
  novel factorized diffusion models to generate natural speech in a zero-shot
  way. Specifically, 1) we design a neural codec with factorized vector
  quantization (FVQ) to disentangle speech waveform into subspaces of content,
  prosody, timbre, and acoustic details; 2) we propose a factorized diffusion
  model to generate attributes in each subspace following its corresponding
  prompt. With this factorization design, NaturalSpeech 3 can effectively and
  efficiently model the intricate speech with disentangled subspaces in a
  divide-and-conquer way. Experiments show that NaturalSpeech 3 outperforms the
  state-of-the-art TTS systems on quality, similarity, prosody, and
  intelligibility. Furthermore, we achieve better performance by scaling to 1B
  parameters and 200K hours of training data.
paper_authors._id: 65e7f92674ab027493c2f417
paper_authors.name: Zeqian Ju
paper_authors._id: 65e7f92674ab027493c2f418
paper_authors.name: Yuancheng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T09:44:10.273Z
paper_authors._id: 6539cc2d48b006575d3f5da2
paper_authors.avatarUrl: /avatars/a2ac1973800e8bb6a72e35fefb436a27.svg
paper_authors.fullname: wangyuancheng
paper_authors.type: user
paper_authors.user: CXKyyds199
paper_authors._id: 65e7f92674ab027493c2f419
paper_authors.name: Kai Shen
paper_authors._id: 65e7f92674ab027493c2f41a
paper_authors.name: Xu Tan
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-05T05:51:26.298Z
paper_authors._id: 5f1040b6e9d71719e3be71d2
paper_authors.avatarUrl: /avatars/a2f28940236ae625ed3810ad62e343ff.svg
paper_authors.fullname: Xu Tan
paper_authors.type: user
paper_authors.user: xutan
paper_authors._id: 65e7f92674ab027493c2f41b
paper_authors.name: Detai Xin
paper_authors._id: 65e7f92674ab027493c2f41c
paper_authors.name: Dongchao Yang
paper_authors._id: 65e7f92674ab027493c2f41d
paper_authors.name: Yanqing Liu
paper_authors._id: 65e7f92674ab027493c2f41e
paper_authors.name: Yichong Leng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T09:56:12.761Z
paper_authors._id: 64a0347b528a9bbe59d6e08c
paper_authors.avatarUrl: /avatars/6dd0bad84d711d1048a0a4169e621773.svg
paper_authors.fullname: Yichong Leng
paper_authors.type: user
paper_authors.user: ustcscallion
paper_authors._id: 65e7f92674ab027493c2f41f
paper_authors.name: Kaitao Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T09:56:19.656Z
paper_authors._id: 5fc0b2b61160c47d1d438568
paper_authors.avatarUrl: /avatars/90beea6b452c662d579197dbf592423a.svg
paper_authors.fullname: Kaitao Song
paper_authors.type: user
paper_authors.user: KaitaoSong
paper_authors._id: 65e7f92674ab027493c2f420
paper_authors.name: Siliang Tang
paper_authors._id: 65e7f92674ab027493c2f421
paper_authors.name: Zhizheng Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-05T06:56:14.830Z
paper_authors._id: 63b4dcefa50cfcefdaa121f3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b4dcefa50cfcefdaa121f3/MlIxOaTSCbZARVyo8Ly7r.jpeg
paper_authors.fullname: Dr Wuz
paper_authors.type: user
paper_authors.user: drwuz
paper_authors._id: 65e7f92674ab027493c2f422
paper_authors.name: Tao Qin
paper_authors._id: 65e7f92674ab027493c2f423
paper_authors.name: Xiang-Yang Li
paper_authors._id: 65e7f92674ab027493c2f424
paper_authors.name: Wei Ye
paper_authors._id: 65e7f92674ab027493c2f425
paper_authors.name: Shikun Zhang
paper_authors._id: 65e7f92674ab027493c2f426
paper_authors.name: Jiang Bian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T09:58:06.909Z
paper_authors._id: 63f253f8f4e30ffd2bd308fb
paper_authors.avatarUrl: /avatars/303f4c7ee588f638acf78a7966786e1e.svg
paper_authors.fullname: Jiang Bian
paper_authors.type: user
paper_authors.user: bianjiang
paper_authors._id: 65e7f92674ab027493c2f427
paper_authors.name: Lei He
paper_authors._id: 65e7f92674ab027493c2f428
paper_authors.name: Jinyu Li
paper_authors._id: 65e7f92674ab027493c2f429
paper_authors.name: Sheng Zhao
paper_title: NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and
    Diffusion Models
paper_summary: While recent large-scale text-to-speech (TTS) models have achieved
  significant progress, they still fall short in speech quality, similarity, and
  prosody. Considering speech intricately encompasses various attributes (e.g.,
  content, prosody, timbre, and acoustic details) that pose significant
  challenges for generation, a natural idea is to factorize speech into
  individual subspaces representing different attributes and generate them
  individually. Motivated by it, we propose NaturalSpeech 3, a TTS system with
  novel factorized diffusion models to generate natural speech in a zero-shot
  way. Specifically, 1) we design a neural codec with factorized vector
  quantization (FVQ) to disentangle speech waveform into subspaces of content,
  prosody, timbre, and acoustic details; 2) we propose a factorized diffusion
  model to generate attributes in each subspace following its corresponding
  prompt. With this factorization design, NaturalSpeech 3 can effectively and
  efficiently model the intricate speech with disentangled subspaces in a
  divide-and-conquer way. Experiments show that NaturalSpeech 3 outperforms the
  state-of-the-art TTS systems on quality, similarity, prosody, and
  intelligibility. Furthermore, we achieve better performance by scaling to 1B
  parameters and 200K hours of training data.
paper_ai_keywords: factorized diffusion models neural codec factorized vector quantization (FVQ) disentangled subspaces divide-and-conquer way