title: Zero-shot Cross-lingual Voice Transfer for TTS
summary: In this paper, we introduce a zero-shot Voice Transfer (VT) module that can
  be seamlessly integrated into a multi-lingual Text-to-speech (TTS) system to
  transfer an individual's voice across languages. Our proposed VT module
  comprises a speaker-encoder that processes reference speech, a bottleneck
  layer, and residual adapters, connected to preexisting TTS layers. We compare
  the performance of various configurations of these components and report Mean
  Opinion Score (MOS) and Speaker Similarity across languages. Using a single
  English reference speech per speaker, we achieve an average voice transfer
  similarity score of 73% across nine target languages. Vocal characteristics
  contribute significantly to the construction and perception of individual
  identity. The loss of one's voice, due to physical or neurological conditions,
  can lead to a profound sense of loss, impacting one's core identity. As a case
  study, we demonstrate that our approach can not only transfer typical speech
  but also restore the voices of individuals with dysarthria, even when only
  atypical speech samples are available - a valuable utility for those who have
  never had typical speech or banked their voice. Cross-lingual typical audio
  samples, plus videos demonstrating voice restoration for dysarthric speakers
  are available here
  (google.github.io/tacotron/publications/zero_shot_voice_transfer).
paper_authors._id: 66f2408f51dca07c005bf987
paper_authors.name: Fadi Biadsy
paper_authors._id: 66f2408f51dca07c005bf988
paper_authors.name: Youzheng Chen
paper_authors._id: 66f2408f51dca07c005bf989
paper_authors.name: Isaac Elias
paper_authors._id: 66f2408f51dca07c005bf98a
paper_authors.name: Kyle Kastner
paper_authors._id: 66f2408f51dca07c005bf98b
paper_authors.name: Gary Wang
paper_authors._id: 66f2408f51dca07c005bf98c
paper_authors.name: Andrew Rosenberg
paper_authors._id: 66f2408f51dca07c005bf98d
paper_authors.name: Bhuvana Ramabhadran
paper_title: Zero-shot Cross-lingual Voice Transfer for TTS
paper_summary: In this paper, we introduce a zero-shot Voice Transfer (VT) module that can
  be seamlessly integrated into a multi-lingual Text-to-speech (TTS) system to
  transfer an individual's voice across languages. Our proposed VT module
  comprises a speaker-encoder that processes reference speech, a bottleneck
  layer, and residual adapters, connected to preexisting TTS layers. We compare
  the performance of various configurations of these components and report Mean
  Opinion Score (MOS) and Speaker Similarity across languages. Using a single
  English reference speech per speaker, we achieve an average voice transfer
  similarity score of 73% across nine target languages. Vocal characteristics
  contribute significantly to the construction and perception of individual
  identity. The loss of one's voice, due to physical or neurological conditions,
  can lead to a profound sense of loss, impacting one's core identity. As a case
  study, we demonstrate that our approach can not only transfer typical speech
  but also restore the voices of individuals with dysarthria, even when only
  atypical speech samples are available - a valuable utility for those who have
  never had typical speech or banked their voice. Cross-lingual typical audio
  samples, plus videos demonstrating voice restoration for dysarthric speakers
  are available here
  (google.github.io/tacotron/publications/zero_shot_voice_transfer).
paper_ai_summary: The zero-shot Voice Transfer module integrates into a multi-lingual TTS system to transfer a speaker's voice across languages using a speaker-encoder, bottleneck layer, and residual adapters.
paper_ai_keywords: zero-shot Voice Transfer VT module Text-to-speech TTS speaker-encoder bottleneck layer residual adapters Mean Opinion Score MOS Speaker Similarity voice transfer cross-lingual dysarthria