Sample-Efficient Diffusion for Text-To-Speech Synthesis
<|reference_start|>Sample-Efficient Diffusion for Text-To-Speech Synthesis: This work introduces Sample-Efficient Speech Diffusion (SESD), an algorithm for effective speech synthesis in modest data regimes through latent diffusion. It is based on a novel diffusion architecture, that we call U-Audio Transformer (U-AT), that efficiently scales to long sequences and operates in the latent space of a pre-trained audio autoencoder. Conditioned on character-aware language model representations, SESD achieves impressive results despite training on less than 1k hours of speech - far less than current state-of-the-art systems. In fact, it synthesizes more intelligible speech than the state-of-the-art auto-regressive model, VALL-E, while using less than 2% the training data.<|reference_end|>
@article{lovelace2024sample-efficient,
    title={Sample-Efficient Diffusion for Text-To-Speech Synthesis},
    author={Justin Lovelace, Soham Ray, Kwangyoun Kim, Kilian Q. Weinberger, Felix
  Wu},
    journal={arXiv preprint arXiv:2409.03717},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.03717},
    primaryClass={cs.SD cs.AI cs.LG}
}