Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and
  Attribute Control
Current face reenactment and swapping methods mainly rely on GAN frameworks,
but recent focus has shifted to pre-trained diffusion models for their superior
generation capabilities. However, training these models is resource-intensive,
and the results have not yet achieved satisfactory performance levels. To
address this issue, we introduce Face-Adapter, an efficient and effective
adapter designed for high-precision and high-fidelity face editing for
pre-trained diffusion models. We observe that both face reenactment/swapping
tasks essentially involve combinations of target structure, ID and attribute.
We aim to sufficiently decouple the control of these factors to achieve both
tasks in one model. Specifically, our method contains: 1) A Spatial Condition
Generator that provides precise landmarks and background; 2) A Plug-and-play
Identity Encoder that transfers face embeddings to the text space by a
transformer decoder. 3) An Attribute Controller that integrates spatial
conditions and detailed attributes. Face-Adapter achieves comparable or even
superior performance in terms of motion control precision, ID retention
capability, and generation quality compared to fully fine-tuned face
reenactment/swapping models. Additionally, Face-Adapter seamlessly integrates
with various StableDiffusion models.
664d76149ebdbf50c0cdf6a3
Yue Han
664d76149ebdbf50c0cdf6a4
Junwei Zhu
admin_assigned
2024-05-22T07:47:54.181Z
65aadb2b9aba49e1d03cfccc
/avatars/b7c48dc999bf2f94018315b546d50eeb.svg
Junwei Zhu
user
ktpt123
664d76149ebdbf50c0cdf6a5
Keke He
admin_assigned
2024-05-22T07:48:00.535Z
6321933bb97c618f9a5b3d8c
/avatars/7742dab4eee85acbdce111b5dd6deceb.svg
kekehehe
user
kekehehe
664d76149ebdbf50c0cdf6a6
Xu Chen
664d76149ebdbf50c0cdf6a7
Yanhao Ge
admin_assigned
2024-05-22T07:48:16.958Z
630d5be915433862cfce3d15
/avatars/f35bc753c6cb954aead2c40de8ccbdbc.svg
Yanhao Ge
user
halege
664d76149ebdbf50c0cdf6a8
Wei Li
664d76149ebdbf50c0cdf6a9
Xiangtai Li
admin_assigned
2024-05-22T07:49:11.798Z
63958b4414513eaf9029ebf1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
Xiangtai Li
user
LXT
664d76149ebdbf50c0cdf6aa
Jiangning Zhang
664d76149ebdbf50c0cdf6ab
Chengjie Wang
664d76149ebdbf50c0cdf6ac
Yong Liu
Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and
  Attribute Control
Current face reenactment and swapping methods mainly rely on GAN frameworks,
but recent focus has shifted to pre-trained diffusion models for their superior
generation capabilities. However, training these models is resource-intensive,
and the results have not yet achieved satisfactory performance levels. To
address this issue, we introduce Face-Adapter, an efficient and effective
adapter designed for high-precision and high-fidelity face editing for
pre-trained diffusion models. We observe that both face reenactment/swapping
tasks essentially involve combinations of target structure, ID and attribute.
We aim to sufficiently decouple the control of these factors to achieve both
tasks in one model. Specifically, our method contains: 1) A Spatial Condition
Generator that provides precise landmarks and background; 2) A Plug-and-play
Identity Encoder that transfers face embeddings to the text space by a
transformer decoder. 3) An Attribute Controller that integrates spatial
conditions and detailed attributes. Face-Adapter achieves comparable or even
superior performance in terms of motion control precision, ID retention
capability, and generation quality compared to fully fine-tuned face
reenactment/swapping models. Additionally, Face-Adapter seamlessly integrates
with various StableDiffusion models.
Face-Adapter, an adaptive framework for pre-trained diffusion models, achieves high-fidelity face editing with efficient identity and attribute control, demonstrating performance competitive with fully fine-tuned models in face reenactment and swapping.
GAN diffusion models Face-Adapter Spatial Condition Generator Plug-and-play Identity Encoder Attribute Controller motion control ID retention generation quality StableDiffusion