title: Adaptive Guidance: Training-free Acceleration of Conditional Diffusion
    Models
summary: This paper presents a comprehensive study on the role of Classifier-Free
  Guidance (CFG) in text-conditioned diffusion models from the perspective of
  inference efficiency. In particular, we relax the default choice of applying
  CFG in all diffusion steps and instead search for efficient guidance policies.
  We formulate the discovery of such policies in the differentiable Neural
  Architecture Search framework. Our findings suggest that the denoising steps
  proposed by CFG become increasingly aligned with simple conditional steps,
  which renders the extra neural network evaluation of CFG redundant, especially
  in the second half of the denoising process. Building upon this insight, we
  propose "Adaptive Guidance" (AG), an efficient variant of CFG, that adaptively
  omits network evaluations when the denoising process displays convergence. Our
  experiments demonstrate that AG preserves CFG's image quality while reducing
  computation by 25%. Thus, AG constitutes a plug-and-play alternative to
  Guidance Distillation, achieving 50% of the speed-ups of the latter while being
  training-free and retaining the capacity to handle negative prompts. Finally,
  we uncover further redundancies of CFG in the first half of the diffusion
  process, showing that entire neural function evaluations can be replaced by
  simple affine transformations of past score estimates. This method, termed
  LinearAG, offers even cheaper inference at the cost of deviating from the
  baseline model. Our findings provide insights into the efficiency of the
  conditional denoising process that contribute to more practical and swift
  deployment of text-conditioned diffusion models.
paper_authors._id: 6583c29bc73f74776b0405bd
paper_authors.name: Angela Castillo
paper_authors._id: 6583c29bc73f74776b0405be
paper_authors.name: Jonas Kohler
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T09:26:41.710Z
paper_authors._id: 6408a3a19e9f790c905281c2
paper_authors.avatarUrl: /avatars/3517962e54bda141018e13f7e21fb1ae.svg
paper_authors.fullname: jonas köhler
paper_authors.type: user
paper_authors.user: Jonaskohler
paper_authors._id: 6583c29bc73f74776b0405bf
paper_authors.name: Juan C. Pérez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T09:26:48.982Z
paper_authors._id: 650bd843a74f76a6ac1f21b1
paper_authors.avatarUrl: /avatars/9d5e549e0b289e77bac8ffbca4f60add.svg
paper_authors.fullname: Juan C. Pérez
paper_authors.type: user
paper_authors.user: juancprzs
paper_authors._id: 6583c29bc73f74776b0405c0
paper_authors.name: Juan Pablo Pérez
paper_authors._id: 6583c29bc73f74776b0405c1
paper_authors.name: Albert Pumarola
paper_authors._id: 6583c29bc73f74776b0405c2
paper_authors.name: Bernard Ghanem
paper_authors._id: 6583c29bc73f74776b0405c3
paper_authors.name: Pablo Arbeláez
paper_authors._id: 6583c29bc73f74776b0405c4
paper_authors.name: Ali Thabet
paper_title: Adaptive Guidance: Training-free Acceleration of Conditional Diffusion
    Models
paper_summary: This paper presents a comprehensive study on the role of Classifier-Free
  Guidance (CFG) in text-conditioned diffusion models from the perspective of
  inference efficiency. In particular, we relax the default choice of applying
  CFG in all diffusion steps and instead search for efficient guidance policies.
  We formulate the discovery of such policies in the differentiable Neural
  Architecture Search framework. Our findings suggest that the denoising steps
  proposed by CFG become increasingly aligned with simple conditional steps,
  which renders the extra neural network evaluation of CFG redundant, especially
  in the second half of the denoising process. Building upon this insight, we
  propose "Adaptive Guidance" (AG), an efficient variant of CFG, that adaptively
  omits network evaluations when the denoising process displays convergence. Our
  experiments demonstrate that AG preserves CFG's image quality while reducing
  computation by 25%. Thus, AG constitutes a plug-and-play alternative to
  Guidance Distillation, achieving 50% of the speed-ups of the latter while being
  training-free and retaining the capacity to handle negative prompts. Finally,
  we uncover further redundancies of CFG in the first half of the diffusion
  process, showing that entire neural function evaluations can be replaced by
  simple affine transformations of past score estimates. This method, termed
  LinearAG, offers even cheaper inference at the cost of deviating from the
  baseline model. Our findings provide insights into the efficiency of the
  conditional denoising process that contribute to more practical and swift
  deployment of text-conditioned diffusion models.
paper_ai_summary: Classifier-Free Guidance (CFG) in text-conditioned diffusion models is analyzed and optimized, leading to the development of Adaptive Guidance (AG) and LinearAG, which reduce computational cost while maintaining image quality.
paper_ai_keywords: Classifier-Free Guidance diffusion models Neural Architecture Search denoising steps Adaptive Guidance LinearAG negative prompts conditional denoising process inference efficiency