VividFace: A Diffusion-Based Hybrid Framework for High-Fidelity Video
  Face Swapping
Video face swapping is becoming increasingly popular across various
applications, yet existing methods primarily focus on static images and
struggle with video face swapping because of temporal consistency and complex
scenarios. In this paper, we present the first diffusion-based framework
specifically designed for video face swapping. Our approach introduces a novel
image-video hybrid training framework that leverages both abundant static image
data and temporal video sequences, addressing the inherent limitations of
video-only training. The framework incorporates a specially designed diffusion
model coupled with a VidFaceVAE that effectively processes both types of data
to better maintain temporal coherence of the generated videos. To further
disentangle identity and pose features, we construct the Attribute-Identity
Disentanglement Triplet (AIDT) Dataset, where each triplet has three face
images, with two images sharing the same pose and two sharing the same
identity. Enhanced with a comprehensive occlusion augmentation, this dataset
also improves robustness against occlusions. Additionally, we integrate 3D
reconstruction techniques as input conditioning to our network for handling
large pose variations. Extensive experiments demonstrate that our framework
achieves superior performance in identity preservation, temporal consistency,
and visual quality compared to existing methods, while requiring fewer
inference steps. Our approach effectively mitigates key challenges in video
face swapping, including temporal flickering, identity preservation, and
robustness to occlusions and pose variations.
676196b4ea3649f2ad086a49
Hao Shao
claimed_verified
2024-12-18T09:52:55.429Z
651bda4337fecec1fe8bb09f
/avatars/f1b3d2b9874022e64d8e5dfeac0e7120.svg
Hao Shao
user
deepcs233
676196b4ea3649f2ad086a4a
Shulun Wang
676196b4ea3649f2ad086a4b
Yang Zhou
676196b4ea3649f2ad086a4c
Guanglu Song
676196b4ea3649f2ad086a4d
Dailan He
676196b4ea3649f2ad086a4e
Shuo Qin
676196b4ea3649f2ad086a4f
Zhuofan Zong
claimed_verified
2024-12-19T18:03:46.713Z
64db05f23725f8d9a9f566c4
/avatars/345b6d2dfadb91ac1f02be023dd27c8f.svg
Zhuofan Zong
user
zongzhuofan
676196b4ea3649f2ad086a50
Bingqi Ma
676196b4ea3649f2ad086a51
Yu Liu
676196b4ea3649f2ad086a52
Hongsheng Li
VividFace: A Diffusion-Based Hybrid Framework for High-Fidelity Video
  Face Swapping
Video face swapping is becoming increasingly popular across various
applications, yet existing methods primarily focus on static images and
struggle with video face swapping because of temporal consistency and complex
scenarios. In this paper, we present the first diffusion-based framework
specifically designed for video face swapping. Our approach introduces a novel
image-video hybrid training framework that leverages both abundant static image
data and temporal video sequences, addressing the inherent limitations of
video-only training. The framework incorporates a specially designed diffusion
model coupled with a VidFaceVAE that effectively processes both types of data
to better maintain temporal coherence of the generated videos. To further
disentangle identity and pose features, we construct the Attribute-Identity
Disentanglement Triplet (AIDT) Dataset, where each triplet has three face
images, with two images sharing the same pose and two sharing the same
identity. Enhanced with a comprehensive occlusion augmentation, this dataset
also improves robustness against occlusions. Additionally, we integrate 3D
reconstruction techniques as input conditioning to our network for handling
large pose variations. Extensive experiments demonstrate that our framework
achieves superior performance in identity preservation, temporal consistency,
and visual quality compared to existing methods, while requiring fewer
inference steps. Our approach effectively mitigates key challenges in video
face swapping, including temporal flickering, identity preservation, and
robustness to occlusions and pose variations.
A diffusion-based framework with a VidFaceVAE and Attribute-Identity Disentanglement Triplet Dataset improves video face swapping through enhanced temporal consistency, identity preservation, and robustness to occlusions and pose variations.
diffusion-based framework diffusion model VidFaceVAE Attribute-Identity Disentanglement Triplet Dataset temporal coherence identity preservation temporal consistency robustness occlusions 3D reconstruction techniques