title: Faster Diffusion: Rethinking the Role of UNet Encoder in Diffusion
    Models
summary: One of the key components within diffusion models is the UNet for noise
  prediction. While several works have explored basic properties of the UNet
  decoder, its encoder largely remains unexplored. In this work, we conduct the
  first comprehensive study of the UNet encoder. We empirically analyze the
  encoder features and provide insights to important questions regarding their
  changes at the inference process. In particular, we find that encoder features
  change gently, whereas the decoder features exhibit substantial variations
  across different time-steps. This finding inspired us to omit the encoder at
  certain adjacent time-steps and reuse cyclically the encoder features in the
  previous time-steps for the decoder. Further based on this observation, we
  introduce a simple yet effective encoder propagation scheme to accelerate the
  diffusion sampling for a diverse set of tasks. By benefiting from our
  propagation scheme, we are able to perform in parallel the decoder at certain
  adjacent time-steps. Additionally, we introduce a prior noise injection method
  to improve the texture details in the generated image. Besides the standard
  text-to-image task, we also validate our approach on other tasks:
  text-to-video, personalized generation and reference-guided generation. Without
  utilizing any knowledge distillation technique, our approach accelerates both
  the Stable Diffusion (SD) and the DeepFloyd-IF models sampling by 41% and
  24% respectively, while maintaining high-quality generation performance. Our
  code is available in
  https://github.com/hutaiHang/Faster-Diffusion{FasterDiffusion}.
paper_authors._id: 657fd43a69cfe9b4cd736b64
paper_authors.name: Senmao Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-18T14:30:43.064Z
paper_authors._id: 637e1cf4f09bf2498c543a73
paper_authors.avatarUrl: /avatars/3c0e4e15602a384839bee0c23029f58a.svg
paper_authors.fullname: Senmao Li
paper_authors.type: user
paper_authors.user: senmaonk
paper_authors._id: 657fd43a69cfe9b4cd736b65
paper_authors.name: Taihang Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T11:10:03.301Z
paper_authors._id: 634bec6ca5d4e109c6aa4895
paper_authors.avatarUrl: /avatars/2f55327097b3782f6f4f59041f3b04b5.svg
paper_authors.fullname: hu
paper_authors.type: user
paper_authors.user: taihang
paper_authors._id: 657fd43a69cfe9b4cd736b66
paper_authors.name: Fahad Shahbaz Khan
paper_authors._id: 657fd43a69cfe9b4cd736b67
paper_authors.name: Linxuan Li
paper_authors._id: 657fd43a69cfe9b4cd736b68
paper_authors.name: Shiqi Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T11:10:36.590Z
paper_authors._id: 648ea2a0df53671f33ee01ed
paper_authors.avatarUrl: /avatars/b0a6af7a86f442a710c694a751936922.svg
paper_authors.fullname: Shiqi Yang
paper_authors.type: user
paper_authors.user: yangsq
paper_authors._id: 657fd43a69cfe9b4cd736b69
paper_authors.name: Yaxing Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T11:10:43.809Z
paper_authors._id: 633242c16eae0bb0a01ea9d7
paper_authors.avatarUrl: /avatars/d0c6ac176332a15238eff84cb8832a00.svg
paper_authors.fullname: Yaxing Wang
paper_authors.type: user
paper_authors.user: Yaxing
paper_authors._id: 657fd43a69cfe9b4cd736b6a
paper_authors.name: Ming-Ming Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T11:11:03.184Z
paper_authors._id: 64e496ae0195913c7fa91c66
paper_authors.avatarUrl: /avatars/23f274f0a3b4ef6ded35205df9bfb564.svg
paper_authors.fullname: chengmingming
paper_authors.type: user
paper_authors.user: mingming8688
paper_authors._id: 657fd43a69cfe9b4cd736b6b
paper_authors.name: Jian Yang
paper_title: Faster Diffusion: Rethinking the Role of UNet Encoder in Diffusion
    Models
paper_summary: One of the key components within diffusion models is the UNet for noise
  prediction. While several works have explored basic properties of the UNet
  decoder, its encoder largely remains unexplored. In this work, we conduct the
  first comprehensive study of the UNet encoder. We empirically analyze the
  encoder features and provide insights to important questions regarding their
  changes at the inference process. In particular, we find that encoder features
  change gently, whereas the decoder features exhibit substantial variations
  across different time-steps. This finding inspired us to omit the encoder at
  certain adjacent time-steps and reuse cyclically the encoder features in the
  previous time-steps for the decoder. Further based on this observation, we
  introduce a simple yet effective encoder propagation scheme to accelerate the
  diffusion sampling for a diverse set of tasks. By benefiting from our
  propagation scheme, we are able to perform in parallel the decoder at certain
  adjacent time-steps. Additionally, we introduce a prior noise injection method
  to improve the texture details in the generated image. Besides the standard
  text-to-image task, we also validate our approach on other tasks:
  text-to-video, personalized generation and reference-guided generation. Without
  utilizing any knowledge distillation technique, our approach accelerates both
  the Stable Diffusion (SD) and the DeepFloyd-IF models sampling by 41% and
  24% respectively, while maintaining high-quality generation performance. Our
  code is available in
  https://github.com/hutaiHang/Faster-Diffusion{FasterDiffusion}.
paper_ai_keywords: UNet noise prediction encoder decoder inference process encoder features decoder features time-steps encoder propagation scheme diffusion sampling parallel sampling prior noise injection texture details text-to-image text-to-video personalized generation reference-guided generation knowledge distillation Stable Diffusion (SD) DeepFloyd-IF