BOOT: Data-free Distillation of Denoising Diffusion Models with
  Bootstrapping
Diffusion models have demonstrated excellent potential for generating diverse
images. However, their performance often suffers from slow generation due to
iterative denoising. Knowledge distillation has been recently proposed as a
remedy that can reduce the number of inference steps to one or a few without
significant quality degradation. However, existing distillation methods either
require significant amounts of offline computation for generating synthetic
training data from the teacher model or need to perform expensive online
learning with the help of real data. In this work, we present a novel technique
called BOOT, that overcomes these limitations with an efficient data-free
distillation algorithm. The core idea is to learn a time-conditioned model that
predicts the output of a pre-trained diffusion model teacher given any time
step. Such a model can be efficiently trained based on bootstrapping from two
consecutive sampled steps. Furthermore, our method can be easily adapted to
large-scale text-to-image diffusion models, which are challenging for
conventional methods given the fact that the training sets are often large and
difficult to access. We demonstrate the effectiveness of our approach on
several benchmark datasets in the DDIM setting, achieving comparable generation
quality while being orders of magnitude faster than the diffusion teacher. The
text-to-image results show that the proposed approach is able to handle highly
complex distributions, shedding light on more efficient generative modeling.
64867cc1847934cf025f1736
Jiatao Gu
admin_assigned
2023-06-12T10:14:55.442Z
6164e72d73996c363c52e66d
https://cdn-avatars.huggingface.co/v1/production/uploads/1634002684894-noauth.png
Jiatao Gu
user
thomagram
64867cc1847934cf025f1737
Shuangfei Zhai
claimed_verified
2024-12-13T15:25:21.267Z
675bf04240726aac79ae8e7b
/avatars/8ae95255dfce91a6f5e76f14eae9e9e5.svg
Shuangfei Zhai
user
shuangfei
64867cc1847934cf025f1738
Yizhe Zhang
admin_assigned
2023-06-12T10:15:48.531Z
5e64858c87403103f9f1055d
https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/96-oNO2OLVkAFOojMaV0T.jpeg
Yizhe Zhang
user
yizhezhang
64867cc1847934cf025f1739
Lingjie Liu
64867cc1847934cf025f173a
Josh Susskind
claimed_verified
2023-06-12T09:10:23.749Z
6470d2247fd7ecdbd0ec3cc9
/avatars/52c5eca12499a1aa9bd49c43d4f20685.svg
Joshua M. Susskind
user
jsusskind
BOOT: Data-free Distillation of Denoising Diffusion Models with
  Bootstrapping
Diffusion models have demonstrated excellent potential for generating diverse
images. However, their performance often suffers from slow generation due to
iterative denoising. Knowledge distillation has been recently proposed as a
remedy that can reduce the number of inference steps to one or a few without
significant quality degradation. However, existing distillation methods either
require significant amounts of offline computation for generating synthetic
training data from the teacher model or need to perform expensive online
learning with the help of real data. In this work, we present a novel technique
called BOOT, that overcomes these limitations with an efficient data-free
distillation algorithm. The core idea is to learn a time-conditioned model that
predicts the output of a pre-trained diffusion model teacher given any time
step. Such a model can be efficiently trained based on bootstrapping from two
consecutive sampled steps. Furthermore, our method can be easily adapted to
large-scale text-to-image diffusion models, which are challenging for
conventional methods given the fact that the training sets are often large and
difficult to access. We demonstrate the effectiveness of our approach on
several benchmark datasets in the DDIM setting, achieving comparable generation
quality while being orders of magnitude faster than the diffusion teacher. The
text-to-image results show that the proposed approach is able to handle highly
complex distributions, shedding light on more efficient generative modeling.
A novel data-free distillation technique called BOOT accelerates diffusion models by learning a time-conditioned predictor, significantly improving generation speed without quality loss.
diffusion models iterative denoising knowledge distillation data-free distillation time-conditioned model bootstrapping pre-trained diffusion model teacher DDIM generative modeling