Low-Resource Cross-Domain Singing Voice Synthesis via Reduced Self-Supervised Speech Representations
<|reference_start|>Low-Resource Cross-Domain Singing Voice Synthesis via Reduced Self-Supervised Speech Representations: In this paper, we propose a singing voice synthesis model, Karaoker-SSL, that is trained only on text and speech data as a typical multi-speaker acoustic model. It is a low-resource pipeline that does not utilize any singing data end-to-end, since its vocoder is also trained on speech data. Karaoker-SSL is conditioned by self-supervised speech representations in an unsupervised manner. We preprocess these representations by selecting only a subset of their task-correlated dimensions. The conditioning module is indirectly guided to capture style information during training by multi-tasking. This is achieved with a Conformer-based module, which predicts the pitch from the acoustic model's output. Thus, Karaoker-SSL allows singing voice synthesis without reliance on hand-crafted and domain-specific features. There are also no requirements for text alignments or lyrics timestamps. To refine the voice quality, we employ a U-Net discriminator that is conditioned on the target speaker and follows a Diffusion GAN training scheme.<|reference_end|>
@article{kakoulidis2024low-resource,
    title={Low-Resource Cross-Domain Singing Voice Synthesis via Reduced
  Self-Supervised Speech Representations},
    author={Panos Kakoulidis, Nikolaos Ellinas, Georgios Vamvoukakis, Myrsini
  Christidou, Alexandra Vioni, Georgia Maniati, Junkwang Oh, Gunu Jho, Inchul
  Hwang, Pirros Tsiakoulis, Aimilios Chalamandaris},
    journal={arXiv preprint arXiv:2402.01520},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.01520},
    primaryClass={cs.SD cs.LG eess.AS}
}