Object-Driven One-Shot Fine-tuning of Text-to-Image Diffusion with
  Prototypical Embedding
As large-scale text-to-image generation models have made remarkable progress
in the field of text-to-image generation, many fine-tuning methods have been
proposed. However, these models often struggle with novel objects, especially
with one-shot scenarios. Our proposed method aims to address the challenges of
generalizability and fidelity in an object-driven way, using only a single
input image and the object-specific regions of interest. To improve
generalizability and mitigate overfitting, in our paradigm, a prototypical
embedding is initialized based on the object's appearance and its class, before
fine-tuning the diffusion model. And during fine-tuning, we propose a
class-characterizing regularization to preserve prior knowledge of object
classes. To further improve fidelity, we introduce object-specific loss, which
can also use to implant multiple objects. Overall, our proposed object-driven
method for implanting new objects can integrate seamlessly with existing
concepts as well as with high fidelity and generalization. Our method
outperforms several existing works. The code will be released.
65b87996cc235dfda331c3b3
Jianxiang Lu
65b87996cc235dfda331c3b4
Cong Xie
extracted_pending
2024-01-30T04:22:47.958Z
6353ef7a74026d8c3af88db6
/avatars/7b1664493cfcd4e62cbd4fc23b2f2bf6.svg
chloe
user
iue
65b87996cc235dfda331c3b5
Hui Guo
extracted_pending
2024-03-05T03:13:35.430Z
65e68d421da1fd85db9bd44a
/avatars/8f02ebd7a7bdab8386ff6962ab3b90ac.svg
BestPerformanceGuo
user
BestPerformance
Object-Driven One-Shot Fine-tuning of Text-to-Image Diffusion with
  Prototypical Embedding
As large-scale text-to-image generation models have made remarkable progress
in the field of text-to-image generation, many fine-tuning methods have been
proposed. However, these models often struggle with novel objects, especially
with one-shot scenarios. Our proposed method aims to address the challenges of
generalizability and fidelity in an object-driven way, using only a single
input image and the object-specific regions of interest. To improve
generalizability and mitigate overfitting, in our paradigm, a prototypical
embedding is initialized based on the object's appearance and its class, before
fine-tuning the diffusion model. And during fine-tuning, we propose a
class-characterizing regularization to preserve prior knowledge of object
classes. To further improve fidelity, we introduce object-specific loss, which
can also use to implant multiple objects. Overall, our proposed object-driven
method for implanting new objects can integrate seamlessly with existing
concepts as well as with high fidelity and generalization. Our method
outperforms several existing works. The code will be released.
A novel fine-tuning method using prototypes and class-characterizing regularization improves object-driven generalization and fidelity in text-to-image generation models.
fine-tuning prototypical embedding diffusion model class-characterizing regularization object-specific loss