Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment
<|reference_start|>Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignment: Large Language Model (LLM) based text-to-speech (TTS) systems have demonstrated remarkable capabilities in handling large speech datasets and generating natural speech for new speakers. However, LLM-based TTS models are not robust as the generated output can contain repeating words, missing words and mis-aligned speech (referred to as hallucinations or attention errors), especially when the text contains multiple occurrences of the same token. We examine these challenges in an encoder-decoder transformer model and find that certain cross-attention heads in such models implicitly learn the text and speech alignment when trained for predicting speech tokens for a given text. To make the alignment more robust, we propose techniques utilizing CTC loss and attention priors that encourage monotonic cross-attention over the text tokens. Our guided attention training technique does not introduce any new learnable parameters and significantly improves robustness of LLM-based TTS models.<|reference_end|>
@article{neekhara2024improving,
    title={Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic
  Alignment},
    author={Paarth Neekhara, Shehzeen Hussain, Subhankar Ghosh, Jason Li, Rafael
  Valle, Rohan Badlani, Boris Ginsburg},
    journal={arXiv preprint arXiv:2406.17957},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.17957},
    primaryClass={cs.SD cs.AI eess.AS}
}