title: Object-Driven One-Shot Fine-tuning of Text-to-Image Diffusion with
    Prototypical Embedding
summary: As large-scale text-to-image generation models have made remarkable progress
  in the field of text-to-image generation, many fine-tuning methods have been
  proposed. However, these models often struggle with novel objects, especially
  with one-shot scenarios. Our proposed method aims to address the challenges of
  generalizability and fidelity in an object-driven way, using only a single
  input image and the object-specific regions of interest. To improve
  generalizability and mitigate overfitting, in our paradigm, a prototypical
  embedding is initialized based on the object's appearance and its class, before
  fine-tuning the diffusion model. And during fine-tuning, we propose a
  class-characterizing regularization to preserve prior knowledge of object
  classes. To further improve fidelity, we introduce object-specific loss, which
  can also use to implant multiple objects. Overall, our proposed object-driven
  method for implanting new objects can integrate seamlessly with existing
  concepts as well as with high fidelity and generalization. Our method
  outperforms several existing works. The code will be released.
paper_authors._id: 65b87996cc235dfda331c3b3
paper_authors.name: Jianxiang Lu
paper_authors._id: 65b87996cc235dfda331c3b4
paper_authors.name: Cong Xie
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-01-30T04:22:47.958Z
paper_authors._id: 6353ef7a74026d8c3af88db6
paper_authors.avatarUrl: /avatars/7b1664493cfcd4e62cbd4fc23b2f2bf6.svg
paper_authors.fullname: chloe
paper_authors.type: user
paper_authors.user: iue
paper_authors._id: 65b87996cc235dfda331c3b5
paper_authors.name: Hui Guo
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-03-05T03:13:35.430Z
paper_authors._id: 65e68d421da1fd85db9bd44a
paper_authors.avatarUrl: /avatars/8f02ebd7a7bdab8386ff6962ab3b90ac.svg
paper_authors.fullname: BestPerformanceGuo
paper_authors.type: user
paper_authors.user: BestPerformance
paper_title: Object-Driven One-Shot Fine-tuning of Text-to-Image Diffusion with
    Prototypical Embedding
paper_summary: As large-scale text-to-image generation models have made remarkable progress
  in the field of text-to-image generation, many fine-tuning methods have been
  proposed. However, these models often struggle with novel objects, especially
  with one-shot scenarios. Our proposed method aims to address the challenges of
  generalizability and fidelity in an object-driven way, using only a single
  input image and the object-specific regions of interest. To improve
  generalizability and mitigate overfitting, in our paradigm, a prototypical
  embedding is initialized based on the object's appearance and its class, before
  fine-tuning the diffusion model. And during fine-tuning, we propose a
  class-characterizing regularization to preserve prior knowledge of object
  classes. To further improve fidelity, we introduce object-specific loss, which
  can also use to implant multiple objects. Overall, our proposed object-driven
  method for implanting new objects can integrate seamlessly with existing
  concepts as well as with high fidelity and generalization. Our method
  outperforms several existing works. The code will be released.
paper_ai_keywords: prototypical embedding diffusion model class-characterizing regularization object-specific loss object-driven method