Cache Me if You Can: Accelerating Diffusion Models through Block Caching
Diffusion models have recently revolutionized the field of image synthesis
due to their ability to generate photorealistic images. However, one of the
major drawbacks of diffusion models is that the image generation process is
costly. A large image-to-image network has to be applied many times to
iteratively refine an image from random noise. While many recent works propose
techniques to reduce the number of required steps, they generally treat the
underlying denoising network as a black box. In this work, we investigate the
behavior of the layers within the network and find that 1) the layers' output
changes smoothly over time, 2) the layers show distinct patterns of change, and
3) the change from step to step is often very small. We hypothesize that many
layer computations in the denoising network are redundant. Leveraging this, we
introduce block caching, in which we reuse outputs from layer blocks of
previous steps to speed up inference. Furthermore, we propose a technique to
automatically determine caching schedules based on each block's changes over
timesteps. In our experiments, we show through FID, human evaluation and
qualitative analysis that Block Caching allows to generate images with higher
visual quality at the same computational cost. We demonstrate this for
different state-of-the-art models (LDM and EMU) and solvers (DDIM and DPM).
65712d87e5739cd97b85f172
Felix Wimbauer
extracted_pending
2025-05-06T14:43:19.143Z
65b11dacb9efde518e3dcbec
/avatars/7fd0476a333d399297163252eec8fa21.svg
Felix Wimbauer
user
fwimbauer
65712d87e5739cd97b85f173
Bichen Wu
admin_assigned
2023-12-07T11:49:28.326Z
6363749520c2cf37bccd9e84
/avatars/e9de7911e67090d883f15a7a58f18e4e.svg
Bichen Wu
user
bichenwu
65712d87e5739cd97b85f174
Edgar Schoenfeld
claimed_verified
2023-12-07T14:02:21.899Z
6571c662d5c6a6d3b0bdae88
/avatars/8080bcdf8b331f62383b724050189660.svg
Edgar Schoenfeld
user
edgarschoenfeld
65712d87e5739cd97b85f175
Xiaoliang Dai
admin_assigned
2023-12-07T11:49:42.916Z
6549417b3ce45eb764faf993
/avatars/d310f475d0697f5f13b3d4141ea0ccaf.svg
Xiaoliang Dai
user
daixl1992
65712d87e5739cd97b85f176
Ji Hou
admin_assigned
2023-12-07T11:50:02.040Z
63084636f0dc38fb47c2dc6b
/avatars/04edff10a048b5d93981e6f5af3ecf40.svg
Ji Hou
user
jhou90
65712d87e5739cd97b85f177
Zijian He
65712d87e5739cd97b85f178
Artsiom Sanakoyeu
claimed_verified
2023-12-12T16:24:14.384Z
62f8e05278e7b6fc437d9ce5
/avatars/618b4a84cd93ac25f00de0616d16da5d.svg
A A
user
Asanakoy
65712d87e5739cd97b85f179
Peizhao Zhang
admin_assigned
2023-12-07T11:50:49.207Z
6433a635a5aed21dd110381e
/avatars/b1fa4fe1c8bd7989875233820fa425af.svg
Peizhao Zhang
user
stzpz
65712d87e5739cd97b85f17a
Sam Tsai
claimed_verified
2024-10-18T15:31:29.924Z
67127b753a058517519fabe8
/avatars/6fabb1634efb8ad8a8a2ae96fd6cf1c6.svg
Sam Tsai
user
sstsai-adl
65712d87e5739cd97b85f17b
Jonas Kohler
admin_assigned
2023-12-07T11:51:09.495Z
6408a3a19e9f790c905281c2
/avatars/3517962e54bda141018e13f7e21fb1ae.svg
jonas köhler
user
Jonaskohler
65712d87e5739cd97b85f17c
Christian Rupprecht
65712d87e5739cd97b85f17d
Daniel Cremers
65712d87e5739cd97b85f17e
Peter Vajda
admin_assigned
2023-12-07T11:51:27.547Z
636f4b35af79dd193603342b
/avatars/577328195738d92ac41a3d0ff3108d7c.svg
Peter Vajda
user
vaphab
65712d87e5739cd97b85f17f
Jialiang Wang
admin_assigned
2023-12-07T11:51:34.843Z
65300c8a5da070af779cf047
/avatars/8e24a2a1d7a8ced05a4c8e25b3d1be5a.svg
Jialiang Wang
user
jwhuggingf
Cache Me if You Can: Accelerating Diffusion Models through Block Caching
Diffusion models have recently revolutionized the field of image synthesis
due to their ability to generate photorealistic images. However, one of the
major drawbacks of diffusion models is that the image generation process is
costly. A large image-to-image network has to be applied many times to
iteratively refine an image from random noise. While many recent works propose
techniques to reduce the number of required steps, they generally treat the
underlying denoising network as a black box. In this work, we investigate the
behavior of the layers within the network and find that 1) the layers' output
changes smoothly over time, 2) the layers show distinct patterns of change, and
3) the change from step to step is often very small. We hypothesize that many
layer computations in the denoising network are redundant. Leveraging this, we
introduce block caching, in which we reuse outputs from layer blocks of
previous steps to speed up inference. Furthermore, we propose a technique to
automatically determine caching schedules based on each block's changes over
timesteps. In our experiments, we show through FID, human evaluation and
qualitative analysis that Block Caching allows to generate images with higher
visual quality at the same computational cost. We demonstrate this for
different state-of-the-art models (LDM and EMU) and solvers (DDIM and DPM).
Block caching in diffusion models improves image generation quality without increasing computational cost by recycling layer outputs.
diffusion models image synthesis denoising network block caching FID human evaluation LDM EMU DDIM DPM