  Prosody transfer is well-studied in the context of expressive speech
synthesis. Cross-lingual prosody transfer, however, is challenging and has been
under-explored to date. In this paper, we present a novel solution to learn
prosody representations that are transferable across languages and speakers for
machine dubbing of expressive multimedia contents. Multimedia contents often
contain field recordings. To enable prosody transfer from noisy audios, we
introduce a novel noise modelling module that disentangles noise conditioning
from prosody conditioning, and thereby gains independent control of noise
levels in the synthesised speech. We augment noisy training data with clean
data to improve the ability of the model to map the denoised reference audio to
clean speech. Our proposed system can generate speech with context-matching
prosody and closes the gap between a strong baseline and human expressive
dialogs by 11.2%.

Audio and Speech Processing