title: Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and
    Attribute Control
summary: Current face reenactment and swapping methods mainly rely on GAN frameworks,
  but recent focus has shifted to pre-trained diffusion models for their superior
  generation capabilities. However, training these models is resource-intensive,
  and the results have not yet achieved satisfactory performance levels. To
  address this issue, we introduce Face-Adapter, an efficient and effective
  adapter designed for high-precision and high-fidelity face editing for
  pre-trained diffusion models. We observe that both face reenactment/swapping
  tasks essentially involve combinations of target structure, ID and attribute.
  We aim to sufficiently decouple the control of these factors to achieve both
  tasks in one model. Specifically, our method contains: 1) A Spatial Condition
  Generator that provides precise landmarks and background; 2) A Plug-and-play
  Identity Encoder that transfers face embeddings to the text space by a
  transformer decoder. 3) An Attribute Controller that integrates spatial
  conditions and detailed attributes. Face-Adapter achieves comparable or even
  superior performance in terms of motion control precision, ID retention
  capability, and generation quality compared to fully fine-tuned face
  reenactment/swapping models. Additionally, Face-Adapter seamlessly integrates
  with various StableDiffusion models.
paper_authors._id: 664d76149ebdbf50c0cdf6a3
paper_authors.name: Yue Han
paper_authors._id: 664d76149ebdbf50c0cdf6a4
paper_authors.name: Junwei Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-22T07:47:54.181Z
paper_authors._id: 65aadb2b9aba49e1d03cfccc
paper_authors.avatarUrl: /avatars/b7c48dc999bf2f94018315b546d50eeb.svg
paper_authors.fullname: Junwei Zhu
paper_authors.type: user
paper_authors.user: ktpt123
paper_authors._id: 664d76149ebdbf50c0cdf6a5
paper_authors.name: Keke He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-22T07:48:00.535Z
paper_authors._id: 6321933bb97c618f9a5b3d8c
paper_authors.avatarUrl: /avatars/7742dab4eee85acbdce111b5dd6deceb.svg
paper_authors.fullname: kekehehe
paper_authors.type: user
paper_authors.user: kekehehe
paper_authors._id: 664d76149ebdbf50c0cdf6a6
paper_authors.name: Xu Chen
paper_authors._id: 664d76149ebdbf50c0cdf6a7
paper_authors.name: Yanhao Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-22T07:48:16.958Z
paper_authors._id: 630d5be915433862cfce3d15
paper_authors.avatarUrl: /avatars/f35bc753c6cb954aead2c40de8ccbdbc.svg
paper_authors.fullname: Yanhao Ge
paper_authors.type: user
paper_authors.user: halege
paper_authors._id: 664d76149ebdbf50c0cdf6a8
paper_authors.name: Wei Li
paper_authors._id: 664d76149ebdbf50c0cdf6a9
paper_authors.name: Xiangtai Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-22T07:49:11.798Z
paper_authors._id: 63958b4414513eaf9029ebf1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
paper_authors.fullname: Xiangtai Li
paper_authors.type: user
paper_authors.user: LXT
paper_authors._id: 664d76149ebdbf50c0cdf6aa
paper_authors.name: Jiangning Zhang
paper_authors._id: 664d76149ebdbf50c0cdf6ab
paper_authors.name: Chengjie Wang
paper_authors._id: 664d76149ebdbf50c0cdf6ac
paper_authors.name: Yong Liu
paper_title: Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and
    Attribute Control
paper_summary: Current face reenactment and swapping methods mainly rely on GAN frameworks,
  but recent focus has shifted to pre-trained diffusion models for their superior
  generation capabilities. However, training these models is resource-intensive,
  and the results have not yet achieved satisfactory performance levels. To
  address this issue, we introduce Face-Adapter, an efficient and effective
  adapter designed for high-precision and high-fidelity face editing for
  pre-trained diffusion models. We observe that both face reenactment/swapping
  tasks essentially involve combinations of target structure, ID and attribute.
  We aim to sufficiently decouple the control of these factors to achieve both
  tasks in one model. Specifically, our method contains: 1) A Spatial Condition
  Generator that provides precise landmarks and background; 2) A Plug-and-play
  Identity Encoder that transfers face embeddings to the text space by a
  transformer decoder. 3) An Attribute Controller that integrates spatial
  conditions and detailed attributes. Face-Adapter achieves comparable or even
  superior performance in terms of motion control precision, ID retention
  capability, and generation quality compared to fully fine-tuned face
  reenactment/swapping models. Additionally, Face-Adapter seamlessly integrates
  with various StableDiffusion models.
paper_ai_summary: Face-Adapter, an adaptive framework for pre-trained diffusion models, achieves high-fidelity face editing with efficient identity and attribute control, demonstrating performance competitive with fully fine-tuned models in face reenactment and swapping.
paper_ai_keywords: GAN diffusion models Face-Adapter Spatial Condition Generator Plug-and-play Identity Encoder Attribute Controller motion control ID retention generation quality StableDiffusion