title: BOOT: Data-free Distillation of Denoising Diffusion Models with
    Bootstrapping
summary: Diffusion models have demonstrated excellent potential for generating diverse
  images. However, their performance often suffers from slow generation due to
  iterative denoising. Knowledge distillation has been recently proposed as a
  remedy that can reduce the number of inference steps to one or a few without
  significant quality degradation. However, existing distillation methods either
  require significant amounts of offline computation for generating synthetic
  training data from the teacher model or need to perform expensive online
  learning with the help of real data. In this work, we present a novel technique
  called BOOT, that overcomes these limitations with an efficient data-free
  distillation algorithm. The core idea is to learn a time-conditioned model that
  predicts the output of a pre-trained diffusion model teacher given any time
  step. Such a model can be efficiently trained based on bootstrapping from two
  consecutive sampled steps. Furthermore, our method can be easily adapted to
  large-scale text-to-image diffusion models, which are challenging for
  conventional methods given the fact that the training sets are often large and
  difficult to access. We demonstrate the effectiveness of our approach on
  several benchmark datasets in the DDIM setting, achieving comparable generation
  quality while being orders of magnitude faster than the diffusion teacher. The
  text-to-image results show that the proposed approach is able to handle highly
  complex distributions, shedding light on more efficient generative modeling.
paper_authors._id: 64867cc1847934cf025f1736
paper_authors.name: Jiatao Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T10:14:55.442Z
paper_authors._id: 6164e72d73996c363c52e66d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1634002684894-noauth.png
paper_authors.fullname: Jiatao Gu
paper_authors.type: user
paper_authors.user: thomagram
paper_authors._id: 64867cc1847934cf025f1737
paper_authors.name: Shuangfei Zhai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:25:21.267Z
paper_authors._id: 675bf04240726aac79ae8e7b
paper_authors.avatarUrl: /avatars/8ae95255dfce91a6f5e76f14eae9e9e5.svg
paper_authors.fullname: Shuangfei Zhai
paper_authors.type: user
paper_authors.user: shuangfei
paper_authors._id: 64867cc1847934cf025f1738
paper_authors.name: Yizhe Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T10:15:48.531Z
paper_authors._id: 5e64858c87403103f9f1055d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/96-oNO2OLVkAFOojMaV0T.jpeg
paper_authors.fullname: Yizhe Zhang
paper_authors.type: user
paper_authors.user: yizhezhang
paper_authors._id: 64867cc1847934cf025f1739
paper_authors.name: Lingjie Liu
paper_authors._id: 64867cc1847934cf025f173a
paper_authors.name: Josh Susskind
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-12T09:10:23.749Z
paper_authors._id: 6470d2247fd7ecdbd0ec3cc9
paper_authors.avatarUrl: /avatars/52c5eca12499a1aa9bd49c43d4f20685.svg
paper_authors.fullname: Joshua M. Susskind
paper_authors.type: user
paper_authors.user: jsusskind
paper_title: BOOT: Data-free Distillation of Denoising Diffusion Models with
    Bootstrapping
paper_summary: Diffusion models have demonstrated excellent potential for generating diverse
  images. However, their performance often suffers from slow generation due to
  iterative denoising. Knowledge distillation has been recently proposed as a
  remedy that can reduce the number of inference steps to one or a few without
  significant quality degradation. However, existing distillation methods either
  require significant amounts of offline computation for generating synthetic
  training data from the teacher model or need to perform expensive online
  learning with the help of real data. In this work, we present a novel technique
  called BOOT, that overcomes these limitations with an efficient data-free
  distillation algorithm. The core idea is to learn a time-conditioned model that
  predicts the output of a pre-trained diffusion model teacher given any time
  step. Such a model can be efficiently trained based on bootstrapping from two
  consecutive sampled steps. Furthermore, our method can be easily adapted to
  large-scale text-to-image diffusion models, which are challenging for
  conventional methods given the fact that the training sets are often large and
  difficult to access. We demonstrate the effectiveness of our approach on
  several benchmark datasets in the DDIM setting, achieving comparable generation
  quality while being orders of magnitude faster than the diffusion teacher. The
  text-to-image results show that the proposed approach is able to handle highly
  complex distributions, shedding light on more efficient generative modeling.
paper_ai_keywords: diffusion models iterative denoising knowledge distillation inference steps synthetic training data teacher model data-free distillation time-conditioned model bootstrapping large-scale text-to-image diffusion models DDIM setting generation quality