Multi-modal Adversarial Training for Zero-Shot Voice Cloning
<|reference_start|>Multi-modal Adversarial Training for Zero-Shot Voice Cloning: A text-to-speech (TTS) model trained to reconstruct speech given text tends towards predictions that are close to the average characteristics of a dataset, failing to model the variations that make human speech sound natural. This problem is magnified for zero-shot voice cloning, a task that requires training data with high variance in speaking styles. We build off of recent works which have used Generative Advsarial Networks (GAN) by proposing a Transformer encoder-decoder architecture to conditionally discriminates between real and generated speech features. The discriminator is used in a training pipeline that improves both the acoustic and prosodic features of a TTS model. We introduce our novel adversarial training technique by applying it to a FastSpeech2 acoustic model and training on Libriheavy, a large multi-speaker dataset, for the task of zero-shot voice cloning. Our model achieves improvements over the baseline in terms of speech quality and speaker similarity. Audio examples from our system are available online.<|reference_end|>
@article{janiczek2024multi-modal,
    title={Multi-modal Adversarial Training for Zero-Shot Voice Cloning},
    author={John Janiczek, Dading Chong, Dongyang Dai, Arlo Faria, Chao Wang, Tao
  Wang, Yuzong Liu},
    journal={arXiv preprint arXiv:2408.15916},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.15916},
    primaryClass={eess.AS cs.LG cs.SD}
}