title: ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually
    Synced Facial Performer
summary: Lip-syncing videos with given audio is the foundation for various
  applications including the creation of virtual presenters or performers. While
  recent studies explore high-fidelity lip-sync with different techniques, their
  task-orientated models either require long-term videos for clip-specific
  training or retain visible artifacts. In this paper, we propose a unified and
  effective framework ReSyncer, that synchronizes generalized audio-visual facial
  information. The key design is revisiting and rewiring the Style-based
  generator to efficiently adopt 3D facial dynamics predicted by a principled
  style-injected Transformer. By simply re-configuring the information insertion
  mechanisms within the noise and style space, our framework fuses motion and
  appearance with unified training. Extensive experiments demonstrate that
  ReSyncer not only produces high-fidelity lip-synced videos according to audio,
  but also supports multiple appealing properties that are suitable for creating
  virtual presenters and performers, including fast personalized fine-tuning,
  video-driven lip-syncing, the transfer of speaking styles, and even face
  swapping. Resources can be found at
  https://guanjz20.github.io/projects/ReSyncer.
paper_authors._id: 66b2deb2b80c1931bb19d15b
paper_authors.name: Jiazhi Guan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:28:20.335Z
paper_authors._id: 65e1515707e203bf747ec55c
paper_authors.avatarUrl: /avatars/9c36e8dcd3f5dd70e9f9cad5d765150e.svg
paper_authors.fullname: JIAZHIGUAN
paper_authors.type: user
paper_authors.user: JIAZHIGUAN
paper_authors._id: 66b2deb2b80c1931bb19d15c
paper_authors.name: Zhiliang Xu
paper_authors._id: 66b2deb2b80c1931bb19d15d
paper_authors.name: Hang Zhou
paper_authors._id: 66b2deb2b80c1931bb19d15e
paper_authors.name: Kaisiyuan Wang
paper_authors._id: 66b2deb2b80c1931bb19d15f
paper_authors.name: Shengyi He
paper_authors._id: 66b2deb2b80c1931bb19d160
paper_authors.name: Zhanwang Zhang
paper_authors._id: 66b2deb2b80c1931bb19d161
paper_authors.name: Borong Liang
paper_authors._id: 66b2deb2b80c1931bb19d162
paper_authors.name: Haocheng Feng
paper_authors._id: 66b2deb2b80c1931bb19d163
paper_authors.name: Errui Ding
paper_authors._id: 66b2deb2b80c1931bb19d164
paper_authors.name: Jingtuo Liu
paper_authors._id: 66b2deb2b80c1931bb19d165
paper_authors.name: Jingdong Wang
paper_authors._id: 66b2deb2b80c1931bb19d166
paper_authors.name: Youjian Zhao
paper_authors._id: 66b2deb2b80c1931bb19d167
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:29:51.195Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually
    Synced Facial Performer
paper_summary: Lip-syncing videos with given audio is the foundation for various
  applications including the creation of virtual presenters or performers. While
  recent studies explore high-fidelity lip-sync with different techniques, their
  task-orientated models either require long-term videos for clip-specific
  training or retain visible artifacts. In this paper, we propose a unified and
  effective framework ReSyncer, that synchronizes generalized audio-visual facial
  information. The key design is revisiting and rewiring the Style-based
  generator to efficiently adopt 3D facial dynamics predicted by a principled
  style-injected Transformer. By simply re-configuring the information insertion
  mechanisms within the noise and style space, our framework fuses motion and
  appearance with unified training. Extensive experiments demonstrate that
  ReSyncer not only produces high-fidelity lip-synced videos according to audio,
  but also supports multiple appealing properties that are suitable for creating
  virtual presenters and performers, including fast personalized fine-tuning,
  video-driven lip-syncing, the transfer of speaking styles, and even face
  swapping. Resources can be found at
  https://guanjz20.github.io/projects/ReSyncer.
paper_ai_summary: ReSyncer is a unified framework that uses a style-injected Transformer and a re-configured Style-based generator to produce high-fidelity lip-synced videos with various appealing properties.
paper_ai_keywords: style-based generator style-injected Transformer 3D facial dynamics noise and style space video-driven lip-syncing speaking style transfer face swapping