title: RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D
    Facial Prior-guided Identity Alignment Network
summary: Person-generic audio-driven face generation is a challenging task in computer
  vision. Previous methods have achieved remarkable progress in audio-visual
  synchronization, but there is still a significant gap between current results
  and practical applications. The challenges are two-fold: 1) Preserving unique
  individual traits for achieving high-precision lip synchronization. 2)
  Generating high-quality facial renderings in real-time performance. In this
  paper, we propose a novel generalized audio-driven framework RealTalk, which
  consists of an audio-to-expression transformer and a high-fidelity
  expression-to-face renderer. In the first component, we consider both identity
  and intra-personal variation features related to speaking lip movements. By
  incorporating cross-modal attention on the enriched facial priors, we can
  effectively align lip movements with audio, thus attaining greater precision in
  expression prediction. In the second component, we design a lightweight facial
  identity alignment (FIA) module which includes a lip-shape control structure
  and a face texture reference structure. This novel design allows us to generate
  fine details in real-time, without depending on sophisticated and inefficient
  feature alignment modules. Our experimental results, both quantitative and
  qualitative, on public datasets demonstrate the clear advantages of our method
  in terms of lip-speech synchronization and generation quality. Furthermore, our
  method is efficient and requires fewer computational resources, making it
  well-suited to meet the needs of practical applications.
paper_authors._id: 6684235c02026ea33b4052dc
paper_authors.name: Xiaozhong Ji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:11:50.184Z
paper_authors._id: 666a59bff0d87d9c3b1dd907
paper_authors.avatarUrl: /avatars/4af5a47d78bca525c7ec985a390408a4.svg
paper_authors.fullname: Xiaozhong Ji
paper_authors.type: user
paper_authors.user: xiaozhongji
paper_authors._id: 6684235c02026ea33b4052dd
paper_authors.name: Chuming Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:11:56.391Z
paper_authors._id: 6307176edc6b7663aa918217
paper_authors.avatarUrl: /avatars/b84528fedc607886d3827e10bd23be7b.svg
paper_authors.fullname: chuminglin
paper_authors.type: user
paper_authors.user: chuminglin
paper_authors._id: 6684235c02026ea33b4052de
paper_authors.name: Zhonggan Ding
paper_authors._id: 6684235c02026ea33b4052df
paper_authors.name: Ying Tai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T21:08:48.745Z
paper_authors._id: 65734004769f3ee9bde1af10
paper_authors.avatarUrl: /avatars/d6310ed861972fd691687d8f47413f33.svg
paper_authors.fullname: Ying Tai
paper_authors.type: user
paper_authors.user: yingtai
paper_authors._id: 6684235c02026ea33b4052e0
paper_authors.name: Jian Yang
paper_authors._id: 6684235c02026ea33b4052e1
paper_authors.name: Junwei Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:12:15.077Z
paper_authors._id: 65aadb2b9aba49e1d03cfccc
paper_authors.avatarUrl: /avatars/b7c48dc999bf2f94018315b546d50eeb.svg
paper_authors.fullname: Junwei Zhu
paper_authors.type: user
paper_authors.user: ktpt123
paper_authors._id: 6684235c02026ea33b4052e2
paper_authors.name: Xiaobin Hu
paper_authors._id: 6684235c02026ea33b4052e3
paper_authors.name: Jiangning Zhang
paper_authors._id: 6684235c02026ea33b4052e4
paper_authors.name: Donghao Luo
paper_authors._id: 6684235c02026ea33b4052e5
paper_authors.name: Chengjie Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:13:00.328Z
paper_authors._id: 65729087379284b904a6d81d
paper_authors.avatarUrl: /avatars/0e7968b1d9221d523a54e3ac787e449d.svg
paper_authors.fullname: Chengjie Wang
paper_authors.type: user
paper_authors.user: chengjie-wang
paper_title: RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D
    Facial Prior-guided Identity Alignment Network
paper_summary: Person-generic audio-driven face generation is a challenging task in computer
  vision. Previous methods have achieved remarkable progress in audio-visual
  synchronization, but there is still a significant gap between current results
  and practical applications. The challenges are two-fold: 1) Preserving unique
  individual traits for achieving high-precision lip synchronization. 2)
  Generating high-quality facial renderings in real-time performance. In this
  paper, we propose a novel generalized audio-driven framework RealTalk, which
  consists of an audio-to-expression transformer and a high-fidelity
  expression-to-face renderer. In the first component, we consider both identity
  and intra-personal variation features related to speaking lip movements. By
  incorporating cross-modal attention on the enriched facial priors, we can
  effectively align lip movements with audio, thus attaining greater precision in
  expression prediction. In the second component, we design a lightweight facial
  identity alignment (FIA) module which includes a lip-shape control structure
  and a face texture reference structure. This novel design allows us to generate
  fine details in real-time, without depending on sophisticated and inefficient
  feature alignment modules. Our experimental results, both quantitative and
  qualitative, on public datasets demonstrate the clear advantages of our method
  in terms of lip-speech synchronization and generation quality. Furthermore, our
  method is efficient and requires fewer computational resources, making it
  well-suited to meet the needs of practical applications.
paper_ai_summary: A new audio-driven face generation framework RealTalk uses an audio-to-expression transformer and a high-fidelity expression-to-face renderer for precise lip synchronization and high-quality real-time facial rendering.
paper_ai_keywords: audio-driven framework audio-to-expression transformer high-fidelity expression-to-face renderer cross-modal attention facial priors lip-shape control structure face texture reference structure lip-speech synchronization