FantasyID: Face Knowledge Enhanced ID-Preserving Video Generation
Tuning-free approaches adapting large-scale pre-trained video diffusion
models for identity-preserving text-to-video generation (IPT2V) have gained
popularity recently due to their efficacy and scalability. However, significant
challenges remain to achieve satisfied facial dynamics while keeping the
identity unchanged. In this work, we present a novel tuning-free IPT2V
framework by enhancing face knowledge of the pre-trained video model built on
diffusion transformers (DiT), dubbed FantasyID. Essentially, 3D facial geometry
prior is incorporated to ensure plausible facial structures during video
synthesis. To prevent the model from learning copy-paste shortcuts that simply
replicate reference face across frames, a multi-view face augmentation strategy
is devised to capture diverse 2D facial appearance features, hence increasing
the dynamics over the facial expressions and head poses. Additionally, after
blending the 2D and 3D features as guidance, instead of naively employing
cross-attention to inject guidance cues into DiT layers, a learnable
layer-aware adaptive mechanism is employed to selectively inject the fused
features into each individual DiT layers, facilitating balanced modeling of
identity preservation and motion dynamics. Experimental results validate our
model's superiority over the current tuning-free IPT2V methods.
67b7ed63c5b2d0bd2eb3774d
Yunpeng Zhang
67b7ed63c5b2d0bd2eb3774e
Qiang Wang
claimed_verified
2025-02-25T09:40:55.195Z
653b195c5f1703225b2fd571
/avatars/b7f376225cef6c13952c9c5540dd43be.svg
wangqiang
user
wangqiang9
67b7ed63c5b2d0bd2eb3774f
Fan Jiang
67b7ed63c5b2d0bd2eb37750
Yaqi Fan
67b7ed63c5b2d0bd2eb37751
Mu Xu
67b7ed63c5b2d0bd2eb37752
Yonggang Qi
FantasyID: Face Knowledge Enhanced ID-Preserving Video Generation
Tuning-free approaches adapting large-scale pre-trained video diffusion
models for identity-preserving text-to-video generation (IPT2V) have gained
popularity recently due to their efficacy and scalability. However, significant
challenges remain to achieve satisfied facial dynamics while keeping the
identity unchanged. In this work, we present a novel tuning-free IPT2V
framework by enhancing face knowledge of the pre-trained video model built on
diffusion transformers (DiT), dubbed FantasyID. Essentially, 3D facial geometry
prior is incorporated to ensure plausible facial structures during video
synthesis. To prevent the model from learning copy-paste shortcuts that simply
replicate reference face across frames, a multi-view face augmentation strategy
is devised to capture diverse 2D facial appearance features, hence increasing
the dynamics over the facial expressions and head poses. Additionally, after
blending the 2D and 3D features as guidance, instead of naively employing
cross-attention to inject guidance cues into DiT layers, a learnable
layer-aware adaptive mechanism is employed to selectively inject the fused
features into each individual DiT layers, facilitating balanced modeling of
identity preservation and motion dynamics. Experimental results validate our
model's superiority over the current tuning-free IPT2V methods.
A novel tuning-free IPT2V framework using diffusion transformers with 3D facial geometry prior and layer-aware adaptive mechanism achieves superior identity preservation and facial dynamics.
diffusion transformers DiT identity-preserving text-to-video generation IPT2V 3D facial geometry prior multi-view face augmentation cross-attention layer-aware adaptive mechanism