title: Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesis
summary: We are interested in a novel task, namely low-resource text-to-talking
  avatar. Given only a few-minute-long talking person video with the audio track
  as the training data and arbitrary texts as the driving input, we aim to
  synthesize high-quality talking portrait videos corresponding to the input
  text. This task has broad application prospects in the digital human industry
  but has not been technically achieved yet due to two challenges: (1) It is
  challenging to mimic the timbre from out-of-domain audio for a traditional
  multi-speaker Text-to-Speech system. (2) It is hard to render high-fidelity and
  lip-synchronized talking avatars with limited training data. In this paper, we
  introduce Adaptive Text-to-Talking Avatar (Ada-TTA), which (1) designs a
  generic zero-shot multi-speaker TTS model that well disentangles the text
  content, timbre, and prosody; and (2) embraces recent advances in neural
  rendering to achieve realistic audio-driven talking face video generation. With
  these designs, our method overcomes the aforementioned two challenges and
  achieves to generate identity-preserving speech and realistic talking person
  video. Experiments demonstrate that our method could synthesize realistic,
  identity-preserving, and audio-visual synchronized talking avatar videos.
paper_authors._id: 647fddff9466a3c9ba4ec5c8
paper_authors.name: Zhenhui Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-07T19:39:59.177Z
paper_authors._id: 6277fcd8c5c547c31b80cc77
paper_authors.avatarUrl: /avatars/7dab3f5c879b5c5eb86f822119a9b5bb.svg
paper_authors.fullname: Zhenhui Ye
paper_authors.type: user
paper_authors.user: yerfor
paper_authors._id: 647fddff9466a3c9ba4ec5c9
paper_authors.name: Ziyue Jiang
paper_authors._id: 647fddff9466a3c9ba4ec5ca
paper_authors.name: Yi Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-07T10:10:04.128Z
paper_authors._id: 6188c6aa72735d93d362efc8
paper_authors.avatarUrl: /avatars/35bdd5f859c7fa47062e8153f974c32c.svg
paper_authors.fullname: Yi Ren
paper_authors.type: user
paper_authors.user: RayeRen
paper_authors._id: 647fddff9466a3c9ba4ec5cb
paper_authors.name: Jinglin Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T14:05:49.821Z
paper_authors._id: 6205c6ad00e630c7a5d0fc2e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1645184905991-6205c6ad00e630c7a5d0fc2e.jpeg
paper_authors.fullname: Jinglin Liu
paper_authors.type: user
paper_authors.user: Silentlin
paper_authors._id: 647fddff9466a3c9ba4ec5cc
paper_authors.name: Chen Zhang
paper_authors._id: 647fddff9466a3c9ba4ec5cd
paper_authors.name: Xiang Yin
paper_authors._id: 647fddff9466a3c9ba4ec5ce
paper_authors.name: Zejun Ma
paper_authors._id: 647fddff9466a3c9ba4ec5cf
paper_authors.name: Zhou Zhao
paper_title: Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesis
paper_summary: We are interested in a novel task, namely low-resource text-to-talking
  avatar. Given only a few-minute-long talking person video with the audio track
  as the training data and arbitrary texts as the driving input, we aim to
  synthesize high-quality talking portrait videos corresponding to the input
  text. This task has broad application prospects in the digital human industry
  but has not been technically achieved yet due to two challenges: (1) It is
  challenging to mimic the timbre from out-of-domain audio for a traditional
  multi-speaker Text-to-Speech system. (2) It is hard to render high-fidelity and
  lip-synchronized talking avatars with limited training data. In this paper, we
  introduce Adaptive Text-to-Talking Avatar (Ada-TTA), which (1) designs a
  generic zero-shot multi-speaker TTS model that well disentangles the text
  content, timbre, and prosody; and (2) embraces recent advances in neural
  rendering to achieve realistic audio-driven talking face video generation. With
  these designs, our method overcomes the aforementioned two challenges and
  achieves to generate identity-preserving speech and realistic talking person
  video. Experiments demonstrate that our method could synthesize realistic,
  identity-preserving, and audio-visual synchronized talking avatar videos.
paper_ai_summary: Ada-TTA synthesizes realistic, identity-preserving, and synchronized talking avatars using a generic zero-shot multi-speaker TTS model and advances in neural rendering.
paper_ai_keywords: Text-to-Talking Avatar Adaptive Text-to-Talking Avatar zero-shot multi-speaker TTS model timbre prosody neural rendering audio-driven talking face video generation