Clockwork Diffusion: Efficient Generation With Model-Step Distillation
This work aims to improve the efficiency of text-to-image diffusion models.
While diffusion models use computationally expensive UNet-based denoising
operations in every generation step, we identify that not all operations are
equally relevant for the final output quality. In particular, we observe that
UNet layers operating on high-res feature maps are relatively sensitive to
small perturbations. In contrast, low-res feature maps influence the semantic
layout of the final image and can often be perturbed with no noticeable change
in the output. Based on this observation, we propose Clockwork Diffusion, a
method that periodically reuses computation from preceding denoising steps to
approximate low-res feature maps at one or more subsequent steps. For multiple
baselines, and for both text-to-image generation and image editing, we
demonstrate that Clockwork leads to comparable or improved perceptual scores
with drastically reduced computational complexity. As an example, for Stable
Diffusion v1.5 with 8 DPM++ steps we save 32% of FLOPs with negligible FID and
CLIP change.
657a706153e2fa36fed22574
Amirhossein Habibian
admin_assigned
2023-12-14T13:49:47.937Z
628372aa09aa80237c6d2044
/avatars/c1e34dee217d73f5539a0807adc12c80.svg
Amir Habibian
user
habibian
657a706153e2fa36fed22575
Amir Ghodrati
claimed_verified
2024-12-11T13:39:28.767Z
65d31d1bfe21569868432e96
/avatars/195207d241f4eab121e4fed6cf5d578d.svg
Amir
user
aghodrati
657a706153e2fa36fed22576
Noor Fathima
admin_assigned
2023-12-14T13:49:59.932Z
64f066ac479ac4e6146908c1
/avatars/4dbd00c724039230334abb023f4b41a9.svg
Noor Fathima
user
Noor000
657a706153e2fa36fed22577
Guillaume Sautiere
657a706153e2fa36fed22578
Risheek Garrepalli
657a706153e2fa36fed22579
Fatih Porikli
657a706153e2fa36fed2257a
Jens Petersen
admin_assigned
2023-12-14T13:50:20.322Z
64f066f27a581687ff8806a1
/avatars/36e28ad0a77477dad6973e54cf244d4f.svg
jens petersen
user
JensTheMan
Clockwork Diffusion: Efficient Generation With Model-Step Distillation
This work aims to improve the efficiency of text-to-image diffusion models.
While diffusion models use computationally expensive UNet-based denoising
operations in every generation step, we identify that not all operations are
equally relevant for the final output quality. In particular, we observe that
UNet layers operating on high-res feature maps are relatively sensitive to
small perturbations. In contrast, low-res feature maps influence the semantic
layout of the final image and can often be perturbed with no noticeable change
in the output. Based on this observation, we propose Clockwork Diffusion, a
method that periodically reuses computation from preceding denoising steps to
approximate low-res feature maps at one or more subsequent steps. For multiple
baselines, and for both text-to-image generation and image editing, we
demonstrate that Clockwork leads to comparable or improved perceptual scores
with drastically reduced computational complexity. As an example, for Stable
Diffusion v1.5 with 8 DPM++ steps we save 32% of FLOPs with negligible FID and
CLIP change.
Clockwork Diffusion reduces computational complexity in text-to-image diffusion models by reusing low-resolution feature maps, maintaining perceptual quality.
diffusion models UNet denoising operations high-res feature maps low-res feature maps Clockwork Diffusion perceptual scores FLOPs FID CLIP text-to-image generation image editing DPM++