title: DomainStudio: Fine-Tuning Diffusion Models for Domain-Driven Image
    Generation using Limited Data
summary: Denoising diffusion probabilistic models (DDPMs) have been proven capable of
  synthesizing high-quality images with remarkable diversity when trained on
  large amounts of data. Typical diffusion models and modern large-scale
  conditional generative models like text-to-image generative models are
  vulnerable to overfitting when fine-tuned on extremely limited data. Existing
  works have explored subject-driven generation using a reference set containing
  a few images. However, few prior works explore DDPM-based domain-driven
  generation, which aims to learn the common features of target domains while
  maintaining diversity. This paper proposes a novel DomainStudio approach to
  adapt DDPMs pre-trained on large-scale source datasets to target domains using
  limited data. It is designed to keep the diversity of subjects provided by
  source domains and get high-quality and diverse adapted samples in target
  domains. We propose to keep the relative distances between adapted samples to
  achieve considerable generation diversity. In addition, we further enhance the
  learning of high-frequency details for better generation quality. Our approach
  is compatible with both unconditional and conditional diffusion models. This
  work makes the first attempt to realize unconditional few-shot image generation
  with diffusion models, achieving better quality and greater diversity than
  current state-of-the-art GAN-based approaches. Moreover, this work also
  significantly relieves overfitting for conditional generation and realizes
  high-quality domain-driven generation, further expanding the applicable
  scenarios of modern large-scale text-to-image models.
paper_authors._id: 649a5f88d20c08c84a484e8c
paper_authors.name: Jingyuan Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T20:15:05.859Z
paper_authors._id: 646ce370da8e99940b6e9840
paper_authors.avatarUrl: /avatars/37308189b6c5bfa4bbd428c5981cf59e.svg
paper_authors.fullname: JingYuan Zhu
paper_authors.type: user
paper_authors.user: jy-zhu
paper_authors._id: 649a5f88d20c08c84a484e8d
paper_authors.name: Huimin Ma
paper_authors._id: 649a5f88d20c08c84a484e8e
paper_authors.name: Jiansheng Chen
paper_authors._id: 649a5f88d20c08c84a484e8f
paper_authors.name: Jian Yuan
paper_title: DomainStudio: Fine-Tuning Diffusion Models for Domain-Driven Image
    Generation using Limited Data
paper_summary: Denoising diffusion probabilistic models (DDPMs) have been proven capable of
  synthesizing high-quality images with remarkable diversity when trained on
  large amounts of data. Typical diffusion models and modern large-scale
  conditional generative models like text-to-image generative models are
  vulnerable to overfitting when fine-tuned on extremely limited data. Existing
  works have explored subject-driven generation using a reference set containing
  a few images. However, few prior works explore DDPM-based domain-driven
  generation, which aims to learn the common features of target domains while
  maintaining diversity. This paper proposes a novel DomainStudio approach to
  adapt DDPMs pre-trained on large-scale source datasets to target domains using
  limited data. It is designed to keep the diversity of subjects provided by
  source domains and get high-quality and diverse adapted samples in target
  domains. We propose to keep the relative distances between adapted samples to
  achieve considerable generation diversity. In addition, we further enhance the
  learning of high-frequency details for better generation quality. Our approach
  is compatible with both unconditional and conditional diffusion models. This
  work makes the first attempt to realize unconditional few-shot image generation
  with diffusion models, achieving better quality and greater diversity than
  current state-of-the-art GAN-based approaches. Moreover, this work also
  significantly relieves overfitting for conditional generation and realizes
  high-quality domain-driven generation, further expanding the applicable
  scenarios of modern large-scale text-to-image models.
paper_ai_keywords: denoising diffusion probabilistic models (DDPMs) overfitting text-to-image generative models conditional generative models subject-driven generation DDPM-based domain-driven generation DomainStudio source datasets target domains generation diversity high-frequency details unconditional diffusion models conditional diffusion models few-shot image generation GAN-based approaches domain-driven generation