title: VividFace: A Diffusion-Based Hybrid Framework for High-Fidelity Video
    Face Swapping
summary: Video face swapping is becoming increasingly popular across various
  applications, yet existing methods primarily focus on static images and
  struggle with video face swapping because of temporal consistency and complex
  scenarios. In this paper, we present the first diffusion-based framework
  specifically designed for video face swapping. Our approach introduces a novel
  image-video hybrid training framework that leverages both abundant static image
  data and temporal video sequences, addressing the inherent limitations of
  video-only training. The framework incorporates a specially designed diffusion
  model coupled with a VidFaceVAE that effectively processes both types of data
  to better maintain temporal coherence of the generated videos. To further
  disentangle identity and pose features, we construct the Attribute-Identity
  Disentanglement Triplet (AIDT) Dataset, where each triplet has three face
  images, with two images sharing the same pose and two sharing the same
  identity. Enhanced with a comprehensive occlusion augmentation, this dataset
  also improves robustness against occlusions. Additionally, we integrate 3D
  reconstruction techniques as input conditioning to our network for handling
  large pose variations. Extensive experiments demonstrate that our framework
  achieves superior performance in identity preservation, temporal consistency,
  and visual quality compared to existing methods, while requiring fewer
  inference steps. Our approach effectively mitigates key challenges in video
  face swapping, including temporal flickering, identity preservation, and
  robustness to occlusions and pose variations.
paper_authors._id: 676196b4ea3649f2ad086a49
paper_authors.name: Hao Shao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-18T09:52:55.429Z
paper_authors._id: 651bda4337fecec1fe8bb09f
paper_authors.avatarUrl: /avatars/f1b3d2b9874022e64d8e5dfeac0e7120.svg
paper_authors.fullname: Hao Shao
paper_authors.type: user
paper_authors.user: deepcs233
paper_authors._id: 676196b4ea3649f2ad086a4a
paper_authors.name: Shulun Wang
paper_authors._id: 676196b4ea3649f2ad086a4b
paper_authors.name: Yang Zhou
paper_authors._id: 676196b4ea3649f2ad086a4c
paper_authors.name: Guanglu Song
paper_authors._id: 676196b4ea3649f2ad086a4d
paper_authors.name: Dailan He
paper_authors._id: 676196b4ea3649f2ad086a4e
paper_authors.name: Shuo Qin
paper_authors._id: 676196b4ea3649f2ad086a4f
paper_authors.name: Zhuofan Zong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:03:46.713Z
paper_authors._id: 64db05f23725f8d9a9f566c4
paper_authors.avatarUrl: /avatars/345b6d2dfadb91ac1f02be023dd27c8f.svg
paper_authors.fullname: Zhuofan Zong
paper_authors.type: user
paper_authors.user: zongzhuofan
paper_authors._id: 676196b4ea3649f2ad086a50
paper_authors.name: Bingqi Ma
paper_authors._id: 676196b4ea3649f2ad086a51
paper_authors.name: Yu Liu
paper_authors._id: 676196b4ea3649f2ad086a52
paper_authors.name: Hongsheng Li
paper_title: VividFace: A Diffusion-Based Hybrid Framework for High-Fidelity Video
    Face Swapping
paper_summary: Video face swapping is becoming increasingly popular across various
  applications, yet existing methods primarily focus on static images and
  struggle with video face swapping because of temporal consistency and complex
  scenarios. In this paper, we present the first diffusion-based framework
  specifically designed for video face swapping. Our approach introduces a novel
  image-video hybrid training framework that leverages both abundant static image
  data and temporal video sequences, addressing the inherent limitations of
  video-only training. The framework incorporates a specially designed diffusion
  model coupled with a VidFaceVAE that effectively processes both types of data
  to better maintain temporal coherence of the generated videos. To further
  disentangle identity and pose features, we construct the Attribute-Identity
  Disentanglement Triplet (AIDT) Dataset, where each triplet has three face
  images, with two images sharing the same pose and two sharing the same
  identity. Enhanced with a comprehensive occlusion augmentation, this dataset
  also improves robustness against occlusions. Additionally, we integrate 3D
  reconstruction techniques as input conditioning to our network for handling
  large pose variations. Extensive experiments demonstrate that our framework
  achieves superior performance in identity preservation, temporal consistency,
  and visual quality compared to existing methods, while requiring fewer
  inference steps. Our approach effectively mitigates key challenges in video
  face swapping, including temporal flickering, identity preservation, and
  robustness to occlusions and pose variations.
paper_ai_keywords: diffusion-based framework image-video hybrid training diffusion model VidFaceVAE Attribute-Identity Disentanglement Triplet (AIDT) occlusion augmentation 3D reconstruction techniques temporal coherence identity preservation temporal consistency visual quality temporal flickering