title: Autoregressive Speech Synthesis without Vector Quantization
summary: We present MELLE, a novel continuous-valued tokens based language modeling
  approach for text to speech synthesis (TTS). MELLE autoregressively generates
  continuous mel-spectrogram frames directly from text condition, bypassing the
  need for vector quantization, which are originally designed for audio
  compression and sacrifice fidelity compared to mel-spectrograms. Specifically,
  (i) instead of cross-entropy loss, we apply regression loss with a proposed
  spectrogram flux loss function to model the probability distribution of the
  continuous-valued tokens. (ii) we have incorporated variational inference into
  MELLE to facilitate sampling mechanisms, thereby enhancing the output diversity
  and model robustness. Experiments demonstrate that, compared to the two-stage
  codec language models VALL-E and its variants, the single-stage MELLE mitigates
  robustness issues by avoiding the inherent flaws of sampling discrete codes,
  achieves superior performance across multiple metrics, and, most importantly,
  offers a more streamlined paradigm. See https://aka.ms/melle for demos of our
  work.
paper_authors._id: 66908f8cd01e0949db0befc9
paper_authors.name: Lingwei Meng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T12:38:12.873Z
paper_authors._id: 65d8d2d0182779056299f668
paper_authors.avatarUrl: /avatars/2f21723a6348ee55431336b740aac9d7.svg
paper_authors.fullname: Lingwei Meng
paper_authors.type: user
paper_authors.user: LingweiMeng
paper_authors._id: 66908f8cd01e0949db0befca
paper_authors.name: Long Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T12:38:45.429Z
paper_authors._id: 6254fe4b777c5f644a41e1be
paper_authors.avatarUrl: /avatars/9059897b33cb926710daf852654054ba.svg
paper_authors.fullname: Long Zhou
paper_authors.type: user
paper_authors.user: lozhou
paper_authors._id: 66908f8cd01e0949db0befcb
paper_authors.name: Shujie Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T12:38:52.066Z
paper_authors._id: 64dc94f1bdac91d86834e13b
paper_authors.avatarUrl: /avatars/264ebe3867121f56eefac6e1ab8cf17f.svg
paper_authors.fullname: Shujie Liu
paper_authors.type: user
paper_authors.user: j4ckl1u
paper_authors._id: 66908f8cd01e0949db0befcc
paper_authors.name: Sanyuan Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T12:38:57.262Z
paper_authors._id: 6465843ce27766e8921b9cab
paper_authors.avatarUrl: /avatars/6017db24da593c9afb21edfcedcddfd4.svg
paper_authors.fullname: Sanyuan Chen
paper_authors.type: user
paper_authors.user: Sanyuan-Chen
paper_authors._id: 66908f8cd01e0949db0befcd
paper_authors.name: Bing Han
paper_authors._id: 66908f8cd01e0949db0befce
paper_authors.name: Shujie Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T12:39:18.247Z
paper_authors._id: 62a9ed2e097098dc397b5b82
paper_authors.avatarUrl: /avatars/cb6b05ec338e4f46187151527157781d.svg
paper_authors.fullname: Shujie Hu
paper_authors.type: user
paper_authors.user: sjhu
paper_authors._id: 66908f8cd01e0949db0befcf
paper_authors.name: Yanqing Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T12:39:29.050Z
paper_authors._id: 654a771a2d2fcd6bf27ff0de
paper_authors.avatarUrl: /avatars/6312b23c642009f13c7993ce4658d06d.svg
paper_authors.fullname: Liuyanqing
paper_authors.type: user
paper_authors.user: Yanqing0327
paper_authors._id: 66908f8cd01e0949db0befd0
paper_authors.name: Jinyu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T12:40:30.853Z
paper_authors._id: 644dffbe16703fd6702d3bd5
paper_authors.avatarUrl: /avatars/dab5831f689ffec9cd1f63213539d3f0.svg
paper_authors.fullname: Jinyu Li
paper_authors.type: user
paper_authors.user: JinyuID
paper_authors._id: 66908f8cd01e0949db0befd1
paper_authors.name: Sheng Zhao
paper_authors._id: 66908f8cd01e0949db0befd2
paper_authors.name: Xixin Wu
paper_authors._id: 66908f8cd01e0949db0befd3
paper_authors.name: Helen Meng
paper_authors._id: 66908f8cd01e0949db0befd4
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-12T12:41:20.379Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: Autoregressive Speech Synthesis without Vector Quantization
paper_summary: We present MELLE, a novel continuous-valued tokens based language modeling
  approach for text to speech synthesis (TTS). MELLE autoregressively generates
  continuous mel-spectrogram frames directly from text condition, bypassing the
  need for vector quantization, which are originally designed for audio
  compression and sacrifice fidelity compared to mel-spectrograms. Specifically,
  (i) instead of cross-entropy loss, we apply regression loss with a proposed
  spectrogram flux loss function to model the probability distribution of the
  continuous-valued tokens. (ii) we have incorporated variational inference into
  MELLE to facilitate sampling mechanisms, thereby enhancing the output diversity
  and model robustness. Experiments demonstrate that, compared to the two-stage
  codec language models VALL-E and its variants, the single-stage MELLE mitigates
  robustness issues by avoiding the inherent flaws of sampling discrete codes,
  achieves superior performance across multiple metrics, and, most importantly,
  offers a more streamlined paradigm. See https://aka.ms/melle for demos of our
  work.
paper_ai_keywords: continuous-valued tokens text to speech synthesis TTS autoregressive generation mel-spectrogram frames vector quantization regression loss spectrogram flux loss variational inference sampling mechanisms single-stage model two-stage codec VALL-E output diversity model robustness