title: ID-Aligner: Enhancing Identity-Preserving Text-to-Image Generation with
    Reward Feedback Learning
summary: The rapid development of diffusion models has triggered diverse applications.
  Identity-preserving text-to-image generation (ID-T2I) particularly has received
  significant attention due to its wide range of application scenarios like AI
  portrait and advertising. While existing ID-T2I methods have demonstrated
  impressive results, several key challenges remain: (1) It is hard to maintain
  the identity characteristics of reference portraits accurately, (2) The
  generated images lack aesthetic appeal especially while enforcing identity
  retention, and (3) There is a limitation that cannot be compatible with
  LoRA-based and Adapter-based methods simultaneously. To address these issues,
  we present ID-Aligner, a general feedback learning framework to
  enhance ID-T2I performance. To resolve identity features lost, we introduce
  identity consistency reward fine-tuning to utilize the feedback from face
  detection and recognition models to improve generated identity preservation.
  Furthermore, we propose identity aesthetic reward fine-tuning leveraging
  rewards from human-annotated preference data and automatically constructed
  feedback on character structure generation to provide aesthetic tuning signals.
  Thanks to its universal feedback fine-tuning framework, our method can be
  readily applied to both LoRA and Adapter models, achieving consistent
  performance gains. Extensive experiments on SD1.5 and SDXL diffusion models
  validate the effectiveness of our approach. Project Page:
  \url{https://idaligner.github.io/}
paper_authors._id: 662a7078c9652518e153a734
paper_authors.name: Weifeng Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-26T07:42:54.212Z
paper_authors._id: 62a7fc03ac97233f161c2433
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a7fc03ac97233f161c2433/19wL1aCwyR-ik2WDXvns9.jpeg
paper_authors.fullname: weifeng
paper_authors.type: user
paper_authors.user: weifeng-chen
paper_authors._id: 662a7078c9652518e153a735
paper_authors.name: Jiacheng Zhang
paper_authors._id: 662a7078c9652518e153a736
paper_authors.name: Jie Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-28T19:30:32.789Z
paper_authors._id: 6381c5d63680a7cf34e08ca9
paper_authors.avatarUrl: /avatars/731467e2d80d0ae163c4a00a9e3ff9e5.svg
paper_authors.fullname: wujie10558@gmail.com
paper_authors.type: user
paper_authors.user: wujie10
paper_authors._id: 662a7078c9652518e153a737
paper_authors.name: Hefeng Wu
paper_authors._id: 662a7078c9652518e153a738
paper_authors.name: Xuefeng Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T12:55:23.839Z
paper_authors._id: 646b7f71df2609a541c1ab9f
paper_authors.avatarUrl: /avatars/48b82e5fd9b06f41ff825507c36816cd.svg
paper_authors.fullname: Xuefeng Xiao
paper_authors.type: user
paper_authors.user: xiaoxuefeng
paper_authors._id: 662a7078c9652518e153a739
paper_authors.name: Liang Lin
paper_title: ID-Aligner: Enhancing Identity-Preserving Text-to-Image Generation with
    Reward Feedback Learning
paper_summary: The rapid development of diffusion models has triggered diverse applications.
  Identity-preserving text-to-image generation (ID-T2I) particularly has received
  significant attention due to its wide range of application scenarios like AI
  portrait and advertising. While existing ID-T2I methods have demonstrated
  impressive results, several key challenges remain: (1) It is hard to maintain
  the identity characteristics of reference portraits accurately, (2) The
  generated images lack aesthetic appeal especially while enforcing identity
  retention, and (3) There is a limitation that cannot be compatible with
  LoRA-based and Adapter-based methods simultaneously. To address these issues,
  we present ID-Aligner, a general feedback learning framework to
  enhance ID-T2I performance. To resolve identity features lost, we introduce
  identity consistency reward fine-tuning to utilize the feedback from face
  detection and recognition models to improve generated identity preservation.
  Furthermore, we propose identity aesthetic reward fine-tuning leveraging
  rewards from human-annotated preference data and automatically constructed
  feedback on character structure generation to provide aesthetic tuning signals.
  Thanks to its universal feedback fine-tuning framework, our method can be
  readily applied to both LoRA and Adapter models, achieving consistent
  performance gains. Extensive experiments on SD1.5 and SDXL diffusion models
  validate the effectiveness of our approach. Project Page:
  \url{https://idaligner.github.io/}
paper_ai_summary: ID-Aligner enhances identity-preserving text-to-image generation through feedback learning, addressing challenges in identity retention, aesthetic appeal, and compatibility with parameter-efficient fine-tuning methods.
paper_ai_keywords: diffusion models identity-preserving text-to-image generation ID-T2I identity consistency reward fine-tuning identity aesthetic reward fine-tuning feedback learning LoRA Adapter SD1.5 SDXL