Eliminating Oversaturation and Artifacts of High Guidance Scales in
  Diffusion Models
Classifier-free guidance (CFG) is crucial for improving both generation
quality and alignment between the input condition and final output in diffusion
models. While a high guidance scale is generally required to enhance these
aspects, it also causes oversaturation and unrealistic artifacts. In this
paper, we revisit the CFG update rule and introduce modifications to address
this issue. We first decompose the update term in CFG into parallel and
orthogonal components with respect to the conditional model prediction and
observe that the parallel component primarily causes oversaturation, while the
orthogonal component enhances image quality. Accordingly, we propose
down-weighting the parallel component to achieve high-quality generations
without oversaturation. Additionally, we draw a connection between CFG and
gradient ascent and introduce a new rescaling and momentum method for the CFG
update rule based on this insight. Our approach, termed adaptive projected
guidance (APG), retains the quality-boosting advantages of CFG while enabling
the use of higher guidance scales without oversaturation. APG is easy to
implement and introduces practically no additional computational overhead to
the sampling process. Through extensive experiments, we demonstrate that APG is
compatible with various conditional diffusion models and samplers, leading to
improved FID, recall, and saturation scores while maintaining precision
comparable to CFG, making our method a superior plug-and-play alternative to
standard classifier-free guidance.
66ff8020001816b29eb1082d
Seyedmorteza Sadat
claimed_verified
2024-10-04T07:23:11.988Z
63b4b02a103617b0a5b0ee2e
https://cdn-avatars.huggingface.co/v1/production/uploads/63b4b02a103617b0a5b0ee2e/LasBC-pCnPJ6XuCt6qqcp.jpeg
Seyedmorteza Sadat
user
msadat97
66ff8020001816b29eb1082e
Otmar Hilliges
66ff8020001816b29eb1082f
Romann M. Weber
claimed_verified
2024-10-04T08:35:04.385Z
630f7646197cd3f24e7f8e9f
/avatars/59bbd4ed38277b313051aac78f6808ac.svg
Romann Weber
user
RMW
Eliminating Oversaturation and Artifacts of High Guidance Scales in
  Diffusion Models
Classifier-free guidance (CFG) is crucial for improving both generation
quality and alignment between the input condition and final output in diffusion
models. While a high guidance scale is generally required to enhance these
aspects, it also causes oversaturation and unrealistic artifacts. In this
paper, we revisit the CFG update rule and introduce modifications to address
this issue. We first decompose the update term in CFG into parallel and
orthogonal components with respect to the conditional model prediction and
observe that the parallel component primarily causes oversaturation, while the
orthogonal component enhances image quality. Accordingly, we propose
down-weighting the parallel component to achieve high-quality generations
without oversaturation. Additionally, we draw a connection between CFG and
gradient ascent and introduce a new rescaling and momentum method for the CFG
update rule based on this insight. Our approach, termed adaptive projected
guidance (APG), retains the quality-boosting advantages of CFG while enabling
the use of higher guidance scales without oversaturation. APG is easy to
implement and introduces practically no additional computational overhead to
the sampling process. Through extensive experiments, we demonstrate that APG is
compatible with various conditional diffusion models and samplers, leading to
improved FID, recall, and saturation scores while maintaining precision
comparable to CFG, making our method a superior plug-and-play alternative to
standard classifier-free guidance.
Adaptive projected guidance modifies classifier-free guidance to enhance image quality and reduce oversaturation by decomposing and rescaling update components, enabling higher guidance scales without artifacts.
classifier-free guidance CFG diffusion models conditional model prediction parallel component orthogonal component oversaturation image quality gradient ascent adaptive projected guidance APG FID recall saturation scores precision