title: Step-aware Preference Optimization: Aligning Preference with Denoising
    Performance at Each Step
summary: Recently, Direct Preference Optimization (DPO) has extended its success from
  aligning large language models (LLMs) to aligning text-to-image diffusion
  models with human preferences. Unlike most existing DPO methods that assume all
  diffusion steps share a consistent preference order with the final generated
  images, we argue that this assumption neglects step-specific denoising
  performance and that preference labels should be tailored to each step's
  contribution. To address this limitation, we propose Step-aware Preference
  Optimization (SPO), a novel post-training approach that independently evaluates
  and adjusts the denoising performance at each step, using a step-aware
  preference model and a step-wise resampler to ensure accurate step-aware
  supervision. Specifically, at each denoising step, we sample a pool of images,
  find a suitable win-lose pair, and, most importantly, randomly select a single
  image from the pool to initialize the next denoising step. This step-wise
  resampler process ensures the next win-lose image pair comes from the same
  image, making the win-lose comparison independent of the previous step. To
  assess the preferences at each step, we train a separate step-aware preference
  model that can be applied to both noisy and clean images. Our experiments with
  Stable Diffusion v1.5 and SDXL demonstrate that SPO significantly outperforms
  the latest Diffusion-DPO in aligning generated images with complex, detailed
  prompts and enhancing aesthetics, while also achieving more than 20x times
  faster in training efficiency. Code and model:
  https://rockeycoss.github.io/spo.github.io/
paper_authors._id: 6662709b68009120affc7a1d
paper_authors.name: Zhanhao Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-07T10:36:01.913Z
paper_authors._id: 64218b9b8adc8881b9738059
paper_authors.avatarUrl: /avatars/03b59d24e18a9aebd10925494304b388.svg
paper_authors.fullname: liangzhanhao
paper_authors.type: user
paper_authors.user: Chinho
paper_authors._id: 6662709b68009120affc7a1e
paper_authors.name: Yuhui Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-07T10:36:15.204Z
paper_authors._id: 631f108bb45367a05fe74260
paper_authors.avatarUrl: /avatars/c20da6800b643728062712d9a2771648.svg
paper_authors.fullname: Researcher
paper_authors.type: user
paper_authors.user: YuanYuhui
paper_authors._id: 6662709b68009120affc7a1f
paper_authors.name: Shuyang Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-10T08:20:15.689Z
paper_authors._id: 64c38fcf573c5a427e12cd37
paper_authors.avatarUrl: /avatars/2b9de06f29147ed2c212e920afba0eaf.svg
paper_authors.fullname: cientgu
paper_authors.type: user
paper_authors.user: cientgu
paper_authors._id: 6662709b68009120affc7a20
paper_authors.name: Bohan Chen
paper_authors._id: 6662709b68009120affc7a21
paper_authors.name: Tiankai Hang
paper_authors._id: 6662709b68009120affc7a22
paper_authors.name: Ji Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T18:37:01.644Z
paper_authors._id: 6473768cbf9b32c6bbc9caa6
paper_authors.avatarUrl: /avatars/7df22e0f33aac855a5d2b0e9286c7ad5.svg
paper_authors.fullname: aaa
paper_authors.type: user
paper_authors.user: jiangsuliji
paper_authors._id: 6662709b68009120affc7a23
paper_authors.name: Liang Zheng
paper_title: Step-aware Preference Optimization: Aligning Preference with Denoising
    Performance at Each Step
paper_summary: Recently, Direct Preference Optimization (DPO) has extended its success from
  aligning large language models (LLMs) to aligning text-to-image diffusion
  models with human preferences. Unlike most existing DPO methods that assume all
  diffusion steps share a consistent preference order with the final generated
  images, we argue that this assumption neglects step-specific denoising
  performance and that preference labels should be tailored to each step's
  contribution. To address this limitation, we propose Step-aware Preference
  Optimization (SPO), a novel post-training approach that independently evaluates
  and adjusts the denoising performance at each step, using a step-aware
  preference model and a step-wise resampler to ensure accurate step-aware
  supervision. Specifically, at each denoising step, we sample a pool of images,
  find a suitable win-lose pair, and, most importantly, randomly select a single
  image from the pool to initialize the next denoising step. This step-wise
  resampler process ensures the next win-lose image pair comes from the same
  image, making the win-lose comparison independent of the previous step. To
  assess the preferences at each step, we train a separate step-aware preference
  model that can be applied to both noisy and clean images. Our experiments with
  Stable Diffusion v1.5 and SDXL demonstrate that SPO significantly outperforms
  the latest Diffusion-DPO in aligning generated images with complex, detailed
  prompts and enhancing aesthetics, while also achieving more than 20x times
  faster in training efficiency. Code and model:
  https://rockeycoss.github.io/spo.github.io/
paper_ai_summary: Step-aware Preference Optimization improves alignment of text-to-image diffusion models with human preferences by independent evaluation and adjustment at each denoising step, outperforming existing methods in aesthetics and efficiency.
paper_ai_keywords: Direct Preference Optimization DPO text-to-image diffusion models step-specific denoising performance preference labels Step-aware Preference Optimization SPO post-training approach step-wise resampler step-aware preference model win-lose pair Stable Diffusion SDXL