ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually
  Synced Facial Performer
Lip-syncing videos with given audio is the foundation for various
applications including the creation of virtual presenters or performers. While
recent studies explore high-fidelity lip-sync with different techniques, their
task-orientated models either require long-term videos for clip-specific
training or retain visible artifacts. In this paper, we propose a unified and
effective framework ReSyncer, that synchronizes generalized audio-visual facial
information. The key design is revisiting and rewiring the Style-based
generator to efficiently adopt 3D facial dynamics predicted by a principled
style-injected Transformer. By simply re-configuring the information insertion
mechanisms within the noise and style space, our framework fuses motion and
appearance with unified training. Extensive experiments demonstrate that
ReSyncer not only produces high-fidelity lip-synced videos according to audio,
but also supports multiple appealing properties that are suitable for creating
virtual presenters and performers, including fast personalized fine-tuning,
video-driven lip-syncing, the transfer of speaking styles, and even face
swapping. Resources can be found at
https://guanjz20.github.io/projects/ReSyncer.
66b2deb2b80c1931bb19d15b
Jiazhi Guan
admin_assigned
2024-08-08T12:28:20.335Z
65e1515707e203bf747ec55c
/avatars/9c36e8dcd3f5dd70e9f9cad5d765150e.svg
JIAZHIGUAN
user
JIAZHIGUAN
66b2deb2b80c1931bb19d15c
Zhiliang Xu
66b2deb2b80c1931bb19d15d
Hang Zhou
66b2deb2b80c1931bb19d15e
Kaisiyuan Wang
66b2deb2b80c1931bb19d15f
Shengyi He
66b2deb2b80c1931bb19d160
Zhanwang Zhang
66b2deb2b80c1931bb19d161
Borong Liang
66b2deb2b80c1931bb19d162
Haocheng Feng
66b2deb2b80c1931bb19d163
Errui Ding
66b2deb2b80c1931bb19d164
Jingtuo Liu
66b2deb2b80c1931bb19d165
Jingdong Wang
66b2deb2b80c1931bb19d166
Youjian Zhao
66b2deb2b80c1931bb19d167
Ziwei Liu
admin_assigned
2024-08-08T12:29:51.195Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually
  Synced Facial Performer
Lip-syncing videos with given audio is the foundation for various
applications including the creation of virtual presenters or performers. While
recent studies explore high-fidelity lip-sync with different techniques, their
task-orientated models either require long-term videos for clip-specific
training or retain visible artifacts. In this paper, we propose a unified and
effective framework ReSyncer, that synchronizes generalized audio-visual facial
information. The key design is revisiting and rewiring the Style-based
generator to efficiently adopt 3D facial dynamics predicted by a principled
style-injected Transformer. By simply re-configuring the information insertion
mechanisms within the noise and style space, our framework fuses motion and
appearance with unified training. Extensive experiments demonstrate that
ReSyncer not only produces high-fidelity lip-synced videos according to audio,
but also supports multiple appealing properties that are suitable for creating
virtual presenters and performers, including fast personalized fine-tuning,
video-driven lip-syncing, the transfer of speaking styles, and even face
swapping. Resources can be found at
https://guanjz20.github.io/projects/ReSyncer.
ReSyncer is a unified framework that uses a style-injected Transformer and a re-configured Style-based generator to produce high-fidelity lip-synced videos with various appealing properties.
style-based generator style-injected Transformer 3D facial dynamics noise and style space video-driven lip-syncing speaking style transfer face swapping