title: PolyVoice: Language Models for Speech to Speech Translation
summary: We propose PolyVoice, a language model-based framework for speech-to-speech
  translation (S2ST) system. Our framework consists of two language models: a
  translation language model and a speech synthesis language model. We use
  discretized speech units, which are generated in a fully unsupervised way, and
  thus our framework can be used for unwritten languages. For the speech
  synthesis part, we adopt the existing VALL-E X approach and build a unit-based
  audio language model. This grants our framework the ability to preserve the
  voice characteristics and the speaking style of the original speech. We examine
  our system on Chinese rightarrow English and English rightarrow Spanish
  pairs. Experimental results show that our system can generate speech with high
  translation quality and audio quality. Speech samples are available at
  https://speechtranslation.github.io/polyvoice.
paper_authors._id: 647eab8f18274bce03088d01
paper_authors.name: Qianqian Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-07T10:09:51.551Z
paper_authors._id: 63e31b765dd0688d9363a398
paper_authors.avatarUrl: /avatars/60d92bef43a7d1342f12ef5ac984f025.svg
paper_authors.fullname: Qianqian Dong
paper_authors.type: user
paper_authors.user: QQD
paper_authors._id: 647eab8f18274bce03088d02
paper_authors.name: Zhiying Huang
paper_authors._id: 647eab8f18274bce03088d03
paper_authors.name: Chen Xu
paper_authors._id: 647eab8f18274bce03088d04
paper_authors.name: Yunlong Zhao
paper_authors._id: 647eab8f18274bce03088d05
paper_authors.name: Kexin Wang
paper_authors._id: 647eab8f18274bce03088d06
paper_authors.name: Xuxin Cheng
paper_authors._id: 647eab8f18274bce03088d07
paper_authors.name: Tom Ko
paper_authors._id: 647eab8f18274bce03088d08
paper_authors.name: Qiao Tian
paper_authors._id: 647eab8f18274bce03088d09
paper_authors.name: Tang Li
paper_authors._id: 647eab8f18274bce03088d0a
paper_authors.name: Fengpeng Yue
paper_authors._id: 647eab8f18274bce03088d0b
paper_authors.name: Ye Bai
paper_authors._id: 647eab8f18274bce03088d0c
paper_authors.name: Xi Chen
paper_authors._id: 647eab8f18274bce03088d0d
paper_authors.name: Lu Lu
paper_authors._id: 647eab8f18274bce03088d0e
paper_authors.name: Zejun Ma
paper_authors._id: 647eab8f18274bce03088d0f
paper_authors.name: Yuping Wang
paper_authors._id: 647eab8f18274bce03088d10
paper_authors.name: Mingxuan Wang
paper_authors._id: 647eab8f18274bce03088d11
paper_authors.name: Yuxuan Wang
paper_title: PolyVoice: Language Models for Speech to Speech Translation
paper_summary: We propose PolyVoice, a language model-based framework for speech-to-speech
  translation (S2ST) system. Our framework consists of two language models: a
  translation language model and a speech synthesis language model. We use
  discretized speech units, which are generated in a fully unsupervised way, and
  thus our framework can be used for unwritten languages. For the speech
  synthesis part, we adopt the existing VALL-E X approach and build a unit-based
  audio language model. This grants our framework the ability to preserve the
  voice characteristics and the speaking style of the original speech. We examine
  our system on Chinese rightarrow English and English rightarrow Spanish
  pairs. Experimental results show that our system can generate speech with high
  translation quality and audio quality. Speech samples are available at
  https://speechtranslation.github.io/polyvoice.
paper_ai_summary: PolyVoice, a language model-based framework for speech-to-speech translation, uses discretized speech units and VALL-E X for high-quality voice and style preservation in translation.
paper_ai_keywords: language model-based framework speech-to-speech translation translation language model speech synthesis language model discretized speech units unsupervised generation VALL-E X unit-based audio language model