title: RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting
    for Text-to-Speech Synthesis
summary: We present RALL-E, a robust language modeling method for text-to-speech (TTS)
  synthesis. While previous work based on large language models (LLMs) shows
  impressive performance on zero-shot TTS, such methods often suffer from poor
  robustness, such as unstable prosody (weird pitch and rhythm/duration) and a
  high word error rate (WER), due to the autoregressive prediction style of
  language models. The core idea behind RALL-E is chain-of-thought (CoT)
  prompting, which decomposes the task into simpler steps to enhance the
  robustness of LLM-based TTS. To accomplish this idea, RALL-E first predicts
  prosody features (pitch and duration) of the input text and uses them as
  intermediate conditions to predict speech tokens in a CoT style. Second, RALL-E
  utilizes the predicted duration prompt to guide the computing of self-attention
  weights in Transformer to enforce the model to focus on the corresponding
  phonemes and prosody features when predicting speech tokens. Results of
  comprehensive objective and subjective evaluations demonstrate that, compared
  to a powerful baseline method VALL-E, RALL-E significantly improves the WER of
  zero-shot TTS from 6.3% (without reranking) and 2.1% (with reranking) to
  2.8% and 1.0%, respectively. Furthermore, we demonstrate that RALL-E
  correctly synthesizes sentences that are hard for VALL-E and reduces the error
  rate from 68% to 4%.
paper_authors._id: 660f69515a39fcf64158ed69
paper_authors.name: Detai Xin
paper_authors._id: 660f69515a39fcf64158ed6a
paper_authors.name: Xu Tan
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-05T05:51:36.839Z
paper_authors._id: 5f1040b6e9d71719e3be71d2
paper_authors.avatarUrl: /avatars/a2f28940236ae625ed3810ad62e343ff.svg
paper_authors.fullname: Xu Tan
paper_authors.type: user
paper_authors.user: xutan
paper_authors._id: 660f69515a39fcf64158ed6b
paper_authors.name: Kai Shen
paper_authors._id: 660f69515a39fcf64158ed6c
paper_authors.name: Zeqian Ju
paper_authors._id: 660f69515a39fcf64158ed6d
paper_authors.name: Dongchao Yang
paper_authors._id: 660f69515a39fcf64158ed6e
paper_authors.name: Yuancheng Wang
paper_authors._id: 660f69515a39fcf64158ed6f
paper_authors.name: Shinnosuke Takamichi
paper_authors._id: 660f69515a39fcf64158ed70
paper_authors.name: Hiroshi Saruwatari
paper_authors._id: 660f69515a39fcf64158ed71
paper_authors.name: Shujie Liu
paper_authors._id: 660f69515a39fcf64158ed72
paper_authors.name: Jinyu Li
paper_authors._id: 660f69515a39fcf64158ed73
paper_authors.name: Sheng Zhao
paper_title: RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting
    for Text-to-Speech Synthesis
paper_summary: We present RALL-E, a robust language modeling method for text-to-speech (TTS)
  synthesis. While previous work based on large language models (LLMs) shows
  impressive performance on zero-shot TTS, such methods often suffer from poor
  robustness, such as unstable prosody (weird pitch and rhythm/duration) and a
  high word error rate (WER), due to the autoregressive prediction style of
  language models. The core idea behind RALL-E is chain-of-thought (CoT)
  prompting, which decomposes the task into simpler steps to enhance the
  robustness of LLM-based TTS. To accomplish this idea, RALL-E first predicts
  prosody features (pitch and duration) of the input text and uses them as
  intermediate conditions to predict speech tokens in a CoT style. Second, RALL-E
  utilizes the predicted duration prompt to guide the computing of self-attention
  weights in Transformer to enforce the model to focus on the corresponding
  phonemes and prosody features when predicting speech tokens. Results of
  comprehensive objective and subjective evaluations demonstrate that, compared
  to a powerful baseline method VALL-E, RALL-E significantly improves the WER of
  zero-shot TTS from 6.3% (without reranking) and 2.1% (with reranking) to
  2.8% and 1.0%, respectively. Furthermore, we demonstrate that RALL-E
  correctly synthesizes sentences that are hard for VALL-E and reduces the error
  rate from 68% to 4%.
paper_ai_keywords: chain-of-thought (CoT) prompting prosody features pitch and duration speech tokens self-attention weights Transformer zero-shot TTS word error rate (WER) reranking