title: FantasyID: Face Knowledge Enhanced ID-Preserving Video Generation
summary: Tuning-free approaches adapting large-scale pre-trained video diffusion
  models for identity-preserving text-to-video generation (IPT2V) have gained
  popularity recently due to their efficacy and scalability. However, significant
  challenges remain to achieve satisfied facial dynamics while keeping the
  identity unchanged. In this work, we present a novel tuning-free IPT2V
  framework by enhancing face knowledge of the pre-trained video model built on
  diffusion transformers (DiT), dubbed FantasyID. Essentially, 3D facial geometry
  prior is incorporated to ensure plausible facial structures during video
  synthesis. To prevent the model from learning copy-paste shortcuts that simply
  replicate reference face across frames, a multi-view face augmentation strategy
  is devised to capture diverse 2D facial appearance features, hence increasing
  the dynamics over the facial expressions and head poses. Additionally, after
  blending the 2D and 3D features as guidance, instead of naively employing
  cross-attention to inject guidance cues into DiT layers, a learnable
  layer-aware adaptive mechanism is employed to selectively inject the fused
  features into each individual DiT layers, facilitating balanced modeling of
  identity preservation and motion dynamics. Experimental results validate our
  model's superiority over the current tuning-free IPT2V methods.
paper_authors._id: 67b7ed63c5b2d0bd2eb3774d
paper_authors.name: Yunpeng Zhang
paper_authors._id: 67b7ed63c5b2d0bd2eb3774e
paper_authors.name: Qiang Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-25T09:40:55.195Z
paper_authors._id: 653b195c5f1703225b2fd571
paper_authors.avatarUrl: /avatars/b7f376225cef6c13952c9c5540dd43be.svg
paper_authors.fullname: wangqiang
paper_authors.type: user
paper_authors.user: wangqiang9
paper_authors._id: 67b7ed63c5b2d0bd2eb3774f
paper_authors.name: Fan Jiang
paper_authors._id: 67b7ed63c5b2d0bd2eb37750
paper_authors.name: Yaqi Fan
paper_authors._id: 67b7ed63c5b2d0bd2eb37751
paper_authors.name: Mu Xu
paper_authors._id: 67b7ed63c5b2d0bd2eb37752
paper_authors.name: Yonggang Qi
paper_title: FantasyID: Face Knowledge Enhanced ID-Preserving Video Generation
paper_summary: Tuning-free approaches adapting large-scale pre-trained video diffusion
  models for identity-preserving text-to-video generation (IPT2V) have gained
  popularity recently due to their efficacy and scalability. However, significant
  challenges remain to achieve satisfied facial dynamics while keeping the
  identity unchanged. In this work, we present a novel tuning-free IPT2V
  framework by enhancing face knowledge of the pre-trained video model built on
  diffusion transformers (DiT), dubbed FantasyID. Essentially, 3D facial geometry
  prior is incorporated to ensure plausible facial structures during video
  synthesis. To prevent the model from learning copy-paste shortcuts that simply
  replicate reference face across frames, a multi-view face augmentation strategy
  is devised to capture diverse 2D facial appearance features, hence increasing
  the dynamics over the facial expressions and head poses. Additionally, after
  blending the 2D and 3D features as guidance, instead of naively employing
  cross-attention to inject guidance cues into DiT layers, a learnable
  layer-aware adaptive mechanism is employed to selectively inject the fused
  features into each individual DiT layers, facilitating balanced modeling of
  identity preservation and motion dynamics. Experimental results validate our
  model's superiority over the current tuning-free IPT2V methods.
paper_ai_summary: A novel tuning-free IPT2V framework using diffusion transformers with 3D facial geometry prior and layer-aware adaptive mechanism achieves superior identity preservation and facial dynamics.
paper_ai_keywords: diffusion transformers DiT identity-preserving text-to-video generation IPT2V 3D facial geometry prior multi-view face augmentation cross-attention layer-aware adaptive mechanism