VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editing
<|reference_start|>VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving Zero-Shot Voice Editing: We present VoiceShop, a novel speech-to-speech framework that can modify multiple attributes of speech, such as age, gender, accent, and speech style, in a single forward pass while preserving the input speaker's timbre. Previous works have been constrained to specialized models that can only edit these attributes individually and suffer from the following pitfalls: the magnitude of the conversion effect is weak, there is no zero-shot capability for out-of-distribution speakers, or the synthesized outputs exhibit undesirable timbre leakage. Our work proposes solutions for each of these issues in a simple modular framework based on a conditional diffusion backbone model with optional normalizing flow-based and sequence-to-sequence speaker attribute-editing modules, whose components can be combined or removed during inference to meet a wide array of tasks without additional model finetuning. Audio samples are available at \url{https://voiceshopai.github.io}.<|reference_end|>
@article{anastassiou2024voiceshop:,
    title={VoiceShop: A Unified Speech-to-Speech Framework for Identity-Preserving
  Zero-Shot Voice Editing},
    author={Philip Anastassiou, Zhenyu Tang, Kainan Peng, Dongya Jia, Jiaxin Li,
  Ming Tu, Yuping Wang, Yuxuan Wang, Mingbo Ma},
    journal={arXiv preprint arXiv:2404.06674},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.06674},
    primaryClass={cs.SD cs.AI eess.AS}
}