title: DreamDistribution: Prompt Distribution Learning for Text-to-Image
    Diffusion Models
summary: The popularization of Text-to-Image (T2I) diffusion models enables the
  generation of high-quality images from text descriptions. However, generating
  diverse customized images with reference visual attributes remains challenging.
  This work focuses on personalizing T2I diffusion models at a more abstract
  concept or category level, adapting commonalities from a set of reference
  images while creating new instances with sufficient variations. We introduce a
  solution that allows a pretrained T2I diffusion model to learn a set of soft
  prompts, enabling the generation of novel images by sampling prompts from the
  learned distribution. These prompts offer text-guided editing capabilities and
  additional flexibility in controlling variation and mixing between multiple
  distributions. We also show the adaptability of the learned prompt distribution
  to other tasks, such as text-to-3D. Finally we demonstrate effectiveness of our
  approach through quantitative analysis including automatic evaluation and human
  assessment. Project website: https://briannlongzhao.github.io/DreamDistribution
paper_authors._id: 658a5a2dc4b2004663d2cc7f
paper_authors.name: Brian Nlong Zhao
paper_authors._id: 658a5a2dc4b2004663d2cc80
paper_authors.name: Yuhang Xiao
paper_authors._id: 658a5a2dc4b2004663d2cc81
paper_authors.name: Jiashu Xu
paper_authors._id: 658a5a2dc4b2004663d2cc82
paper_authors.name: Xinyang Jiang
paper_authors._id: 658a5a2dc4b2004663d2cc83
paper_authors.name: Yifan Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:56:06.911Z
paper_authors._id: 62d18eb81e36881a57f29bf4
paper_authors.avatarUrl: /avatars/104851421b4ee9641daaf15942fa7ea1.svg
paper_authors.fullname: Yif Yang
paper_authors.type: user
paper_authors.user: Yif29
paper_authors._id: 658a5a2dc4b2004663d2cc84
paper_authors.name: Dongsheng Li
paper_authors._id: 658a5a2dc4b2004663d2cc85
paper_authors.name: Laurent Itti
paper_authors._id: 658a5a2dc4b2004663d2cc86
paper_authors.name: Vibhav Vineet
paper_authors._id: 658a5a2dc4b2004663d2cc87
paper_authors.name: Yunhao Ge
paper_title: DreamDistribution: Prompt Distribution Learning for Text-to-Image
    Diffusion Models
paper_summary: The popularization of Text-to-Image (T2I) diffusion models enables the
  generation of high-quality images from text descriptions. However, generating
  diverse customized images with reference visual attributes remains challenging.
  This work focuses on personalizing T2I diffusion models at a more abstract
  concept or category level, adapting commonalities from a set of reference
  images while creating new instances with sufficient variations. We introduce a
  solution that allows a pretrained T2I diffusion model to learn a set of soft
  prompts, enabling the generation of novel images by sampling prompts from the
  learned distribution. These prompts offer text-guided editing capabilities and
  additional flexibility in controlling variation and mixing between multiple
  distributions. We also show the adaptability of the learned prompt distribution
  to other tasks, such as text-to-3D. Finally we demonstrate effectiveness of our
  approach through quantitative analysis including automatic evaluation and human
  assessment. Project website: https://briannlongzhao.github.io/DreamDistribution
paper_ai_keywords: Text-to-Image (T2I) diffusion models T2I diffusion models soft prompts prompt distribution text-guided prompting automatic evaluation human assessment text-to-3D