Adaptive Guidance: Training-free Acceleration of Conditional Diffusion
  Models
This paper presents a comprehensive study on the role of Classifier-Free
Guidance (CFG) in text-conditioned diffusion models from the perspective of
inference efficiency. In particular, we relax the default choice of applying
CFG in all diffusion steps and instead search for efficient guidance policies.
We formulate the discovery of such policies in the differentiable Neural
Architecture Search framework. Our findings suggest that the denoising steps
proposed by CFG become increasingly aligned with simple conditional steps,
which renders the extra neural network evaluation of CFG redundant, especially
in the second half of the denoising process. Building upon this insight, we
propose "Adaptive Guidance" (AG), an efficient variant of CFG, that adaptively
omits network evaluations when the denoising process displays convergence. Our
experiments demonstrate that AG preserves CFG's image quality while reducing
computation by 25%. Thus, AG constitutes a plug-and-play alternative to
Guidance Distillation, achieving 50% of the speed-ups of the latter while being
training-free and retaining the capacity to handle negative prompts. Finally,
we uncover further redundancies of CFG in the first half of the diffusion
process, showing that entire neural function evaluations can be replaced by
simple affine transformations of past score estimates. This method, termed
LinearAG, offers even cheaper inference at the cost of deviating from the
baseline model. Our findings provide insights into the efficiency of the
conditional denoising process that contribute to more practical and swift
deployment of text-conditioned diffusion models.
6583c29bc73f74776b0405bd
Angela Castillo
6583c29bc73f74776b0405be
Jonas Kohler
admin_assigned
2023-12-21T09:26:41.710Z
6408a3a19e9f790c905281c2
/avatars/3517962e54bda141018e13f7e21fb1ae.svg
jonas köhler
user
Jonaskohler
6583c29bc73f74776b0405bf
Juan C. Pérez
admin_assigned
2023-12-21T09:26:48.982Z
650bd843a74f76a6ac1f21b1
/avatars/9d5e549e0b289e77bac8ffbca4f60add.svg
Juan C. Pérez
user
juancprzs
6583c29bc73f74776b0405c0
Juan Pablo Pérez
6583c29bc73f74776b0405c1
Albert Pumarola
6583c29bc73f74776b0405c2
Bernard Ghanem
6583c29bc73f74776b0405c3
Pablo Arbeláez
6583c29bc73f74776b0405c4
Ali Thabet
Adaptive Guidance: Training-free Acceleration of Conditional Diffusion
  Models
This paper presents a comprehensive study on the role of Classifier-Free
Guidance (CFG) in text-conditioned diffusion models from the perspective of
inference efficiency. In particular, we relax the default choice of applying
CFG in all diffusion steps and instead search for efficient guidance policies.
We formulate the discovery of such policies in the differentiable Neural
Architecture Search framework. Our findings suggest that the denoising steps
proposed by CFG become increasingly aligned with simple conditional steps,
which renders the extra neural network evaluation of CFG redundant, especially
in the second half of the denoising process. Building upon this insight, we
propose "Adaptive Guidance" (AG), an efficient variant of CFG, that adaptively
omits network evaluations when the denoising process displays convergence. Our
experiments demonstrate that AG preserves CFG's image quality while reducing
computation by 25%. Thus, AG constitutes a plug-and-play alternative to
Guidance Distillation, achieving 50% of the speed-ups of the latter while being
training-free and retaining the capacity to handle negative prompts. Finally,
we uncover further redundancies of CFG in the first half of the diffusion
process, showing that entire neural function evaluations can be replaced by
simple affine transformations of past score estimates. This method, termed
LinearAG, offers even cheaper inference at the cost of deviating from the
baseline model. Our findings provide insights into the efficiency of the
conditional denoising process that contribute to more practical and swift
deployment of text-conditioned diffusion models.
Classifier-Free Guidance (CFG) in text-conditioned diffusion models is analyzed and optimized, leading to the development of Adaptive Guidance (AG) and LinearAG, which reduce computational cost while maintaining image quality.
Classifier-Free Guidance diffusion models Neural Architecture Search denoising steps Adaptive Guidance LinearAG negative prompts conditional denoising process inference efficiency