Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy
  Curvature of Attention
Conditional diffusion models have shown remarkable success in visual content
generation, producing high-quality samples across various domains, largely due
to classifier-free guidance (CFG). Recent attempts to extend guidance to
unconditional models have relied on heuristic techniques, resulting in
suboptimal generation quality and unintended effects. In this work, we propose
Smoothed Energy Guidance (SEG), a novel training- and condition-free approach
that leverages the energy-based perspective of the self-attention mechanism to
enhance image generation. By defining the energy of self-attention, we
introduce a method to reduce the curvature of the energy landscape of attention
and use the output as the unconditional prediction. Practically, we control the
curvature of the energy landscape by adjusting the Gaussian kernel parameter
while keeping the guidance scale parameter fixed. Additionally, we present a
query blurring method that is equivalent to blurring the entire attention
weights without incurring quadratic complexity in the number of tokens. In our
experiments, SEG achieves a Pareto improvement in both quality and the
reduction of side effects. The code is available at
https://github.com/SusungHong/SEG-SDXL.
66acf0d3a3f613f565ef96d7
Susung Hong
claimed_verified
2024-08-02T14:53:46.138Z
635a6dd21668c4ead3ed19fa
https://cdn-avatars.huggingface.co/v1/production/uploads/1679159537671-635a6dd21668c4ead3ed19fa.jpeg
Susung Hong
user
susunghong
Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy
  Curvature of Attention
Conditional diffusion models have shown remarkable success in visual content
generation, producing high-quality samples across various domains, largely due
to classifier-free guidance (CFG). Recent attempts to extend guidance to
unconditional models have relied on heuristic techniques, resulting in
suboptimal generation quality and unintended effects. In this work, we propose
Smoothed Energy Guidance (SEG), a novel training- and condition-free approach
that leverages the energy-based perspective of the self-attention mechanism to
enhance image generation. By defining the energy of self-attention, we
introduce a method to reduce the curvature of the energy landscape of attention
and use the output as the unconditional prediction. Practically, we control the
curvature of the energy landscape by adjusting the Gaussian kernel parameter
while keeping the guidance scale parameter fixed. Additionally, we present a
query blurring method that is equivalent to blurring the entire attention
weights without incurring quadratic complexity in the number of tokens. In our
experiments, SEG achieves a Pareto improvement in both quality and the
reduction of side effects. The code is available at
https://github.com/SusungHong/SEG-SDXL.
Smoothed Energy Guidance (SEG) enhances unconditional image generation by leveraging the self-attention mechanism, achieving better quality and reduced side effects.
conditional diffusion models classifier-free guidance (CFG) Smoothed Energy Guidance (SEG) energy-based perspective self-attention mechanism energy landscape Gaussian kernel parameter query blurring attention weights