title: Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy
    Curvature of Attention
summary: Conditional diffusion models have shown remarkable success in visual content
  generation, producing high-quality samples across various domains, largely due
  to classifier-free guidance (CFG). Recent attempts to extend guidance to
  unconditional models have relied on heuristic techniques, resulting in
  suboptimal generation quality and unintended effects. In this work, we propose
  Smoothed Energy Guidance (SEG), a novel training- and condition-free approach
  that leverages the energy-based perspective of the self-attention mechanism to
  enhance image generation. By defining the energy of self-attention, we
  introduce a method to reduce the curvature of the energy landscape of attention
  and use the output as the unconditional prediction. Practically, we control the
  curvature of the energy landscape by adjusting the Gaussian kernel parameter
  while keeping the guidance scale parameter fixed. Additionally, we present a
  query blurring method that is equivalent to blurring the entire attention
  weights without incurring quadratic complexity in the number of tokens. In our
  experiments, SEG achieves a Pareto improvement in both quality and the
  reduction of side effects. The code is available at
  https://github.com/SusungHong/SEG-SDXL.
paper_authors._id: 66acf0d3a3f613f565ef96d7
paper_authors.name: Susung Hong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-02T14:53:46.138Z
paper_authors._id: 635a6dd21668c4ead3ed19fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1679159537671-635a6dd21668c4ead3ed19fa.jpeg
paper_authors.fullname: Susung Hong
paper_authors.type: user
paper_authors.user: susunghong
paper_title: Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy
    Curvature of Attention
paper_summary: Conditional diffusion models have shown remarkable success in visual content
  generation, producing high-quality samples across various domains, largely due
  to classifier-free guidance (CFG). Recent attempts to extend guidance to
  unconditional models have relied on heuristic techniques, resulting in
  suboptimal generation quality and unintended effects. In this work, we propose
  Smoothed Energy Guidance (SEG), a novel training- and condition-free approach
  that leverages the energy-based perspective of the self-attention mechanism to
  enhance image generation. By defining the energy of self-attention, we
  introduce a method to reduce the curvature of the energy landscape of attention
  and use the output as the unconditional prediction. Practically, we control the
  curvature of the energy landscape by adjusting the Gaussian kernel parameter
  while keeping the guidance scale parameter fixed. Additionally, we present a
  query blurring method that is equivalent to blurring the entire attention
  weights without incurring quadratic complexity in the number of tokens. In our
  experiments, SEG achieves a Pareto improvement in both quality and the
  reduction of side effects. The code is available at
  https://github.com/SusungHong/SEG-SDXL.
paper_ai_keywords: conditional diffusion models classifier-free guidance (CFG) Smoothed Energy Guidance (SEG) energy-based perspective self-attention mechanism energy landscape attention Gaussian kernel parameter query blurring method Pareto improvement