title: Cache Me if You Can: Accelerating Diffusion Models through Block Caching
summary: Diffusion models have recently revolutionized the field of image synthesis
  due to their ability to generate photorealistic images. However, one of the
  major drawbacks of diffusion models is that the image generation process is
  costly. A large image-to-image network has to be applied many times to
  iteratively refine an image from random noise. While many recent works propose
  techniques to reduce the number of required steps, they generally treat the
  underlying denoising network as a black box. In this work, we investigate the
  behavior of the layers within the network and find that 1) the layers' output
  changes smoothly over time, 2) the layers show distinct patterns of change, and
  3) the change from step to step is often very small. We hypothesize that many
  layer computations in the denoising network are redundant. Leveraging this, we
  introduce block caching, in which we reuse outputs from layer blocks of
  previous steps to speed up inference. Furthermore, we propose a technique to
  automatically determine caching schedules based on each block's changes over
  timesteps. In our experiments, we show through FID, human evaluation and
  qualitative analysis that Block Caching allows to generate images with higher
  visual quality at the same computational cost. We demonstrate this for
  different state-of-the-art models (LDM and EMU) and solvers (DDIM and DPM).
paper_authors._id: 65712d87e5739cd97b85f172
paper_authors.name: Felix Wimbauer
paper_authors._id: 65712d87e5739cd97b85f173
paper_authors.name: Bichen Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T11:49:28.326Z
paper_authors._id: 6363749520c2cf37bccd9e84
paper_authors.avatarUrl: /avatars/e9de7911e67090d883f15a7a58f18e4e.svg
paper_authors.fullname: Bichen Wu
paper_authors.type: user
paper_authors.user: bichenwu
paper_authors._id: 65712d87e5739cd97b85f174
paper_authors.name: Edgar Schoenfeld
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-07T14:02:21.899Z
paper_authors._id: 6571c662d5c6a6d3b0bdae88
paper_authors.avatarUrl: /avatars/8080bcdf8b331f62383b724050189660.svg
paper_authors.fullname: Edgar Schoenfeld
paper_authors.type: user
paper_authors.user: edgarschoenfeld
paper_authors._id: 65712d87e5739cd97b85f175
paper_authors.name: Xiaoliang Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T11:49:42.916Z
paper_authors._id: 6549417b3ce45eb764faf993
paper_authors.avatarUrl: /avatars/d310f475d0697f5f13b3d4141ea0ccaf.svg
paper_authors.fullname: Xiaoliang Dai
paper_authors.type: user
paper_authors.user: daixl1992
paper_authors._id: 65712d87e5739cd97b85f176
paper_authors.name: Ji Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T11:50:02.040Z
paper_authors._id: 63084636f0dc38fb47c2dc6b
paper_authors.avatarUrl: /avatars/04edff10a048b5d93981e6f5af3ecf40.svg
paper_authors.fullname: Ji Hou
paper_authors.type: user
paper_authors.user: jhou90
paper_authors._id: 65712d87e5739cd97b85f177
paper_authors.name: Zijian He
paper_authors._id: 65712d87e5739cd97b85f178
paper_authors.name: Artsiom Sanakoyeu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-12T16:24:14.384Z
paper_authors._id: 62f8e05278e7b6fc437d9ce5
paper_authors.avatarUrl: /avatars/618b4a84cd93ac25f00de0616d16da5d.svg
paper_authors.fullname: A A
paper_authors.type: user
paper_authors.user: Asanakoy
paper_authors._id: 65712d87e5739cd97b85f179
paper_authors.name: Peizhao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T11:50:49.207Z
paper_authors._id: 6433a635a5aed21dd110381e
paper_authors.avatarUrl: /avatars/b1fa4fe1c8bd7989875233820fa425af.svg
paper_authors.fullname: Peizhao Zhang
paper_authors.type: user
paper_authors.user: stzpz
paper_authors._id: 65712d87e5739cd97b85f17a
paper_authors.name: Sam Tsai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T15:31:29.924Z
paper_authors._id: 67127b753a058517519fabe8
paper_authors.avatarUrl: /avatars/6fabb1634efb8ad8a8a2ae96fd6cf1c6.svg
paper_authors.fullname: Sam Tsai
paper_authors.type: user
paper_authors.user: sstsai-adl
paper_authors._id: 65712d87e5739cd97b85f17b
paper_authors.name: Jonas Kohler
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T11:51:09.495Z
paper_authors._id: 6408a3a19e9f790c905281c2
paper_authors.avatarUrl: /avatars/3517962e54bda141018e13f7e21fb1ae.svg
paper_authors.fullname: jonas köhler
paper_authors.type: user
paper_authors.user: Jonaskohler
paper_authors._id: 65712d87e5739cd97b85f17c
paper_authors.name: Christian Rupprecht
paper_authors._id: 65712d87e5739cd97b85f17d
paper_authors.name: Daniel Cremers
paper_authors._id: 65712d87e5739cd97b85f17e
paper_authors.name: Peter Vajda
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T11:51:27.547Z
paper_authors._id: 636f4b35af79dd193603342b
paper_authors.avatarUrl: /avatars/577328195738d92ac41a3d0ff3108d7c.svg
paper_authors.fullname: Peter Vajda
paper_authors.type: user
paper_authors.user: vaphab
paper_authors._id: 65712d87e5739cd97b85f17f
paper_authors.name: Jialiang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T11:51:34.843Z
paper_authors._id: 65300c8a5da070af779cf047
paper_authors.avatarUrl: /avatars/8e24a2a1d7a8ced05a4c8e25b3d1be5a.svg
paper_authors.fullname: Jialiang Wang
paper_authors.type: user
paper_authors.user: jwhuggingf
paper_title: Cache Me if You Can: Accelerating Diffusion Models through Block Caching
paper_summary: Diffusion models have recently revolutionized the field of image synthesis
  due to their ability to generate photorealistic images. However, one of the
  major drawbacks of diffusion models is that the image generation process is
  costly. A large image-to-image network has to be applied many times to
  iteratively refine an image from random noise. While many recent works propose
  techniques to reduce the number of required steps, they generally treat the
  underlying denoising network as a black box. In this work, we investigate the
  behavior of the layers within the network and find that 1) the layers' output
  changes smoothly over time, 2) the layers show distinct patterns of change, and
  3) the change from step to step is often very small. We hypothesize that many
  layer computations in the denoising network are redundant. Leveraging this, we
  introduce block caching, in which we reuse outputs from layer blocks of
  previous steps to speed up inference. Furthermore, we propose a technique to
  automatically determine caching schedules based on each block's changes over
  timesteps. In our experiments, we show through FID, human evaluation and
  qualitative analysis that Block Caching allows to generate images with higher
  visual quality at the same computational cost. We demonstrate this for
  different state-of-the-art models (LDM and EMU) and solvers (DDIM and DPM).
paper_ai_keywords: diffusion models image synthesis photorealistic images denoising network layers block caching caching schedules FID human evaluation qualitative analysis LDM EMU DDIM DPM