title: Boosting Large Language Model for Speech Synthesis: An Empirical Study
summary: Large language models (LLMs) have made significant advancements in natural
  language processing and are concurrently extending the language ability to
  other modalities, such as speech and vision. Nevertheless, most of the previous
  work focuses on prompting LLMs with perception abilities like auditory
  comprehension, and the effective approach for augmenting LLMs with speech
  synthesis capabilities remains ambiguous. In this paper, we conduct a
  comprehensive empirical exploration of boosting LLMs with the ability to
  generate speech, by combining pre-trained LLM LLaMA/OPT and text-to-speech
  synthesis model VALL-E. We compare three integration methods between LLMs and
  speech synthesis models, including directly fine-tuned LLMs, superposed layers
  of LLMs and VALL-E, and coupled LLMs and VALL-E using LLMs as a powerful text
  encoder. Experimental results show that, using LoRA method to fine-tune LLMs
  directly to boost the speech synthesis capability does not work well, and
  superposed LLMs and VALL-E can improve the quality of generated speech both in
  speaker similarity and word error rate (WER). Among these three methods,
  coupled methods leveraging LLMs as the text encoder can achieve the best
  performance, making it outperform original speech synthesis models with a
  consistently better speaker similarity and a significant (10.9%) WER reduction.
paper_authors._id: 65939ab054f8826173507d00
paper_authors.name: Hongkun Hao
paper_authors._id: 65939ab054f8826173507d01
paper_authors.name: Long Zhou
paper_authors._id: 65939ab054f8826173507d02
paper_authors.name: Shujie Liu
paper_authors._id: 65939ab054f8826173507d03
paper_authors.name: Jinyu Li
paper_authors._id: 65939ab054f8826173507d04
paper_authors.name: Shujie Hu
paper_authors._id: 65939ab054f8826173507d05
paper_authors.name: Rui Wang
paper_authors._id: 65939ab054f8826173507d06
paper_authors.name: Furu Wei
paper_title: Boosting Large Language Model for Speech Synthesis: An Empirical Study
paper_summary: Large language models (LLMs) have made significant advancements in natural
  language processing and are concurrently extending the language ability to
  other modalities, such as speech and vision. Nevertheless, most of the previous
  work focuses on prompting LLMs with perception abilities like auditory
  comprehension, and the effective approach for augmenting LLMs with speech
  synthesis capabilities remains ambiguous. In this paper, we conduct a
  comprehensive empirical exploration of boosting LLMs with the ability to
  generate speech, by combining pre-trained LLM LLaMA/OPT and text-to-speech
  synthesis model VALL-E. We compare three integration methods between LLMs and
  speech synthesis models, including directly fine-tuned LLMs, superposed layers
  of LLMs and VALL-E, and coupled LLMs and VALL-E using LLMs as a powerful text
  encoder. Experimental results show that, using LoRA method to fine-tune LLMs
  directly to boost the speech synthesis capability does not work well, and
  superposed LLMs and VALL-E can improve the quality of generated speech both in
  speaker similarity and word error rate (WER). Among these three methods,
  coupled methods leveraging LLMs as the text encoder can achieve the best
  performance, making it outperform original speech synthesis models with a
  consistently better speaker similarity and a significant (10.9%) WER reduction.
paper_ai_keywords: large language models (LLMs) natural language processing speech synthesis perception abilities auditory comprehension pre-trained LLM LLaMA/OPT text-to-speech synthesis model VALL-E directly fine-tuned LLMs superposed layers coupled LLMs and VALL-E LoRA method speaker similarity word error rate (WER)