title: Diffusion Self-Distillation for Zero-Shot Customized Image Generation
summary: Text-to-image diffusion models produce impressive results but are frustrating
  tools for artists who desire fine-grained control. For example, a common use
  case is to create images of a specific instance in novel contexts, i.e.,
  "identity-preserving generation". This setting, along with many other tasks
  (e.g., relighting), is a natural fit for image+text-conditional generative
  models. However, there is insufficient high-quality paired data to train such a
  model directly. We propose Diffusion Self-Distillation, a method for using a
  pre-trained text-to-image model to generate its own dataset for
  text-conditioned image-to-image tasks. We first leverage a text-to-image
  diffusion model's in-context generation ability to create grids of images and
  curate a large paired dataset with the help of a Visual-Language Model. We then
  fine-tune the text-to-image model into a text+image-to-image model using the
  curated paired dataset. We demonstrate that Diffusion Self-Distillation
  outperforms existing zero-shot methods and is competitive with per-instance
  tuning techniques on a wide range of identity-preservation generation tasks,
  without requiring test-time optimization.
paper_authors._id: 6747e71ab645f41b0759b4a9
paper_authors.name: Shengqu Cai
paper_authors._id: 6747e71ab645f41b0759b4aa
paper_authors.name: Eric Chan
paper_authors._id: 6747e71ab645f41b0759b4ab
paper_authors.name: Yunzhi Zhang
paper_authors._id: 6747e71ab645f41b0759b4ac
paper_authors.name: Leonidas Guibas
paper_authors._id: 6747e71ab645f41b0759b4ad
paper_authors.name: Jiajun Wu
paper_authors._id: 6747e71ab645f41b0759b4ae
paper_authors.name: Gordon Wetzstein
paper_title: Diffusion Self-Distillation for Zero-Shot Customized Image Generation
paper_summary: Text-to-image diffusion models produce impressive results but are frustrating
  tools for artists who desire fine-grained control. For example, a common use
  case is to create images of a specific instance in novel contexts, i.e.,
  "identity-preserving generation". This setting, along with many other tasks
  (e.g., relighting), is a natural fit for image+text-conditional generative
  models. However, there is insufficient high-quality paired data to train such a
  model directly. We propose Diffusion Self-Distillation, a method for using a
  pre-trained text-to-image model to generate its own dataset for
  text-conditioned image-to-image tasks. We first leverage a text-to-image
  diffusion model's in-context generation ability to create grids of images and
  curate a large paired dataset with the help of a Visual-Language Model. We then
  fine-tune the text-to-image model into a text+image-to-image model using the
  curated paired dataset. We demonstrate that Diffusion Self-Distillation
  outperforms existing zero-shot methods and is competitive with per-instance
  tuning techniques on a wide range of identity-preservation generation tasks,
  without requiring test-time optimization.
paper_ai_summary: Diffusion Self-Distillation uses a pre-trained text-to-image model to generate a dataset for fine-tuning into a text+image-to-image model, improving identity-preservation generation without test-time optimization.
paper_ai_keywords: text-to-image diffusion models identity-preserving generation text-conditioned image-to-image tasks Visual-Language Model Diffusion Self-Distillation zero-shot methods per-instance tuning techniques