Autoregressive Speech Synthesis without Vector Quantization
We present MELLE, a novel continuous-valued tokens based language modeling
approach for text to speech synthesis (TTS). MELLE autoregressively generates
continuous mel-spectrogram frames directly from text condition, bypassing the
need for vector quantization, which are originally designed for audio
compression and sacrifice fidelity compared to mel-spectrograms. Specifically,
(i) instead of cross-entropy loss, we apply regression loss with a proposed
spectrogram flux loss function to model the probability distribution of the
continuous-valued tokens. (ii) we have incorporated variational inference into
MELLE to facilitate sampling mechanisms, thereby enhancing the output diversity
and model robustness. Experiments demonstrate that, compared to the two-stage
codec language models VALL-E and its variants, the single-stage MELLE mitigates
robustness issues by avoiding the inherent flaws of sampling discrete codes,
achieves superior performance across multiple metrics, and, most importantly,
offers a more streamlined paradigm. See https://aka.ms/melle for demos of our
work.
66908f8cd01e0949db0befc9
Lingwei Meng
admin_assigned
2024-07-12T12:38:12.873Z
65d8d2d0182779056299f668
/avatars/2f21723a6348ee55431336b740aac9d7.svg
Lingwei Meng
user
LingweiMeng
66908f8cd01e0949db0befca
Long Zhou
admin_assigned
2024-07-12T12:38:45.429Z
6254fe4b777c5f644a41e1be
/avatars/9059897b33cb926710daf852654054ba.svg
Long Zhou
user
lozhou
66908f8cd01e0949db0befcb
Shujie Liu
admin_assigned
2024-07-12T12:38:52.066Z
64dc94f1bdac91d86834e13b
/avatars/264ebe3867121f56eefac6e1ab8cf17f.svg
Shujie Liu
user
j4ckl1u
66908f8cd01e0949db0befcc
Sanyuan Chen
admin_assigned
2024-07-12T12:38:57.262Z
6465843ce27766e8921b9cab
/avatars/6017db24da593c9afb21edfcedcddfd4.svg
Sanyuan Chen
user
Sanyuan-Chen
66908f8cd01e0949db0befcd
Bing Han
66908f8cd01e0949db0befce
Shujie Hu
admin_assigned
2024-07-12T12:39:18.247Z
62a9ed2e097098dc397b5b82
/avatars/cb6b05ec338e4f46187151527157781d.svg
Shujie Hu
user
sjhu
66908f8cd01e0949db0befcf
Yanqing Liu
admin_assigned
2024-07-12T12:39:29.050Z
654a771a2d2fcd6bf27ff0de
/avatars/6312b23c642009f13c7993ce4658d06d.svg
Liuyanqing
user
Yanqing0327
66908f8cd01e0949db0befd0
Jinyu Li
admin_assigned
2024-07-12T12:40:30.853Z
644dffbe16703fd6702d3bd5
/avatars/dab5831f689ffec9cd1f63213539d3f0.svg
Jinyu Li
user
JinyuID
66908f8cd01e0949db0befd1
Sheng Zhao
66908f8cd01e0949db0befd2
Xixin Wu
66908f8cd01e0949db0befd3
Helen Meng
66908f8cd01e0949db0befd4
Furu Wei
admin_assigned
2024-07-12T12:41:20.379Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
Autoregressive Speech Synthesis without Vector Quantization
We present MELLE, a novel continuous-valued tokens based language modeling
approach for text to speech synthesis (TTS). MELLE autoregressively generates
continuous mel-spectrogram frames directly from text condition, bypassing the
need for vector quantization, which are originally designed for audio
compression and sacrifice fidelity compared to mel-spectrograms. Specifically,
(i) instead of cross-entropy loss, we apply regression loss with a proposed
spectrogram flux loss function to model the probability distribution of the
continuous-valued tokens. (ii) we have incorporated variational inference into
MELLE to facilitate sampling mechanisms, thereby enhancing the output diversity
and model robustness. Experiments demonstrate that, compared to the two-stage
codec language models VALL-E and its variants, the single-stage MELLE mitigates
robustness issues by avoiding the inherent flaws of sampling discrete codes,
achieves superior performance across multiple metrics, and, most importantly,
offers a more streamlined paradigm. See https://aka.ms/melle for demos of our
work.
MELLE, a continuous-valued token-based language model for text-to-speech synthesis, uses regression and variational inference to generate high-fidelity mel-spectrograms directly from text.
continuous-valued tokens language modeling text-to-speech synthesis mel-spectrogram cross-entropy loss regression loss spectrogram flux loss function variational inference VALL-E