title: Clockwork Diffusion: Efficient Generation With Model-Step Distillation
summary: This work aims to improve the efficiency of text-to-image diffusion models.
  While diffusion models use computationally expensive UNet-based denoising
  operations in every generation step, we identify that not all operations are
  equally relevant for the final output quality. In particular, we observe that
  UNet layers operating on high-res feature maps are relatively sensitive to
  small perturbations. In contrast, low-res feature maps influence the semantic
  layout of the final image and can often be perturbed with no noticeable change
  in the output. Based on this observation, we propose Clockwork Diffusion, a
  method that periodically reuses computation from preceding denoising steps to
  approximate low-res feature maps at one or more subsequent steps. For multiple
  baselines, and for both text-to-image generation and image editing, we
  demonstrate that Clockwork leads to comparable or improved perceptual scores
  with drastically reduced computational complexity. As an example, for Stable
  Diffusion v1.5 with 8 DPM++ steps we save 32% of FLOPs with negligible FID and
  CLIP change.
paper_authors._id: 657a706153e2fa36fed22574
paper_authors.name: Amirhossein Habibian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:49:47.937Z
paper_authors._id: 628372aa09aa80237c6d2044
paper_authors.avatarUrl: /avatars/c1e34dee217d73f5539a0807adc12c80.svg
paper_authors.fullname: Amir Habibian
paper_authors.type: user
paper_authors.user: habibian
paper_authors._id: 657a706153e2fa36fed22575
paper_authors.name: Amir Ghodrati
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T13:39:28.767Z
paper_authors._id: 65d31d1bfe21569868432e96
paper_authors.avatarUrl: /avatars/195207d241f4eab121e4fed6cf5d578d.svg
paper_authors.fullname: Amir
paper_authors.type: user
paper_authors.user: aghodrati
paper_authors._id: 657a706153e2fa36fed22576
paper_authors.name: Noor Fathima
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:49:59.932Z
paper_authors._id: 64f066ac479ac4e6146908c1
paper_authors.avatarUrl: /avatars/4dbd00c724039230334abb023f4b41a9.svg
paper_authors.fullname: Noor Fathima
paper_authors.type: user
paper_authors.user: Noor000
paper_authors._id: 657a706153e2fa36fed22577
paper_authors.name: Guillaume Sautiere
paper_authors._id: 657a706153e2fa36fed22578
paper_authors.name: Risheek Garrepalli
paper_authors._id: 657a706153e2fa36fed22579
paper_authors.name: Fatih Porikli
paper_authors._id: 657a706153e2fa36fed2257a
paper_authors.name: Jens Petersen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:50:20.322Z
paper_authors._id: 64f066f27a581687ff8806a1
paper_authors.avatarUrl: /avatars/36e28ad0a77477dad6973e54cf244d4f.svg
paper_authors.fullname: jens petersen
paper_authors.type: user
paper_authors.user: JensTheMan
paper_title: Clockwork Diffusion: Efficient Generation With Model-Step Distillation
paper_summary: This work aims to improve the efficiency of text-to-image diffusion models.
  While diffusion models use computationally expensive UNet-based denoising
  operations in every generation step, we identify that not all operations are
  equally relevant for the final output quality. In particular, we observe that
  UNet layers operating on high-res feature maps are relatively sensitive to
  small perturbations. In contrast, low-res feature maps influence the semantic
  layout of the final image and can often be perturbed with no noticeable change
  in the output. Based on this observation, we propose Clockwork Diffusion, a
  method that periodically reuses computation from preceding denoising steps to
  approximate low-res feature maps at one or more subsequent steps. For multiple
  baselines, and for both text-to-image generation and image editing, we
  demonstrate that Clockwork leads to comparable or improved perceptual scores
  with drastically reduced computational complexity. As an example, for Stable
  Diffusion v1.5 with 8 DPM++ steps we save 32% of FLOPs with negligible FID and
  CLIP change.
paper_ai_keywords: UNet denoising feature maps noise constraint orthogonal Clockwork Diffusion perceptual scores computational complexity FLOPs FID CLIP DPM++