DreamDistribution: Prompt Distribution Learning for Text-to-Image
  Diffusion Models
The popularization of Text-to-Image (T2I) diffusion models enables the
generation of high-quality images from text descriptions. However, generating
diverse customized images with reference visual attributes remains challenging.
This work focuses on personalizing T2I diffusion models at a more abstract
concept or category level, adapting commonalities from a set of reference
images while creating new instances with sufficient variations. We introduce a
solution that allows a pretrained T2I diffusion model to learn a set of soft
prompts, enabling the generation of novel images by sampling prompts from the
learned distribution. These prompts offer text-guided editing capabilities and
additional flexibility in controlling variation and mixing between multiple
distributions. We also show the adaptability of the learned prompt distribution
to other tasks, such as text-to-3D. Finally we demonstrate effectiveness of our
approach through quantitative analysis including automatic evaluation and human
assessment. Project website: https://briannlongzhao.github.io/DreamDistribution
658a5a2dc4b2004663d2cc7f
Brian Nlong Zhao
658a5a2dc4b2004663d2cc80
Yuhang Xiao
658a5a2dc4b2004663d2cc81
Jiashu Xu
658a5a2dc4b2004663d2cc82
Xinyang Jiang
658a5a2dc4b2004663d2cc83
Yifan Yang
claimed_verified
2024-11-15T09:56:06.911Z
62d18eb81e36881a57f29bf4
/avatars/104851421b4ee9641daaf15942fa7ea1.svg
Yif Yang
user
Yif29
658a5a2dc4b2004663d2cc84
Dongsheng Li
658a5a2dc4b2004663d2cc85
Laurent Itti
658a5a2dc4b2004663d2cc86
Vibhav Vineet
658a5a2dc4b2004663d2cc87
Yunhao Ge
DreamDistribution: Prompt Distribution Learning for Text-to-Image
  Diffusion Models
The popularization of Text-to-Image (T2I) diffusion models enables the
generation of high-quality images from text descriptions. However, generating
diverse customized images with reference visual attributes remains challenging.
This work focuses on personalizing T2I diffusion models at a more abstract
concept or category level, adapting commonalities from a set of reference
images while creating new instances with sufficient variations. We introduce a
solution that allows a pretrained T2I diffusion model to learn a set of soft
prompts, enabling the generation of novel images by sampling prompts from the
learned distribution. These prompts offer text-guided editing capabilities and
additional flexibility in controlling variation and mixing between multiple
distributions. We also show the adaptability of the learned prompt distribution
to other tasks, such as text-to-3D. Finally we demonstrate effectiveness of our
approach through quantitative analysis including automatic evaluation and human
assessment. Project website: https://briannlongzhao.github.io/DreamDistribution
A pretrained Text-to-Image diffusion model learns soft prompts to generate diverse images with specific attributes, adapting to text-to-3D and validated through quantitative and qualitative evaluations.
Text-to-Image (T2I) diffusion models soft prompts learned distribution text-guided editing variation control text-to-3D