description: <h1 id="step-aware-preference-optimization:-aligning-preference-with-denoising-performance-at-each-step-wsatoyfe8">Step-aware Preference Optimization: Aligning Preference with Denoising Performance at Each Step</h1><p></p><p><a target="_blank" rel="ugc" href="https://arxiv.org/abs/2406.04314">Arxiv Paper</a></p><p><a target="_blank" rel="ugc" href="https://github.com/RockeyCoss/SPO">Github Code</a></p><p><a target="_blank" rel="ugc" href="https://rockeycoss.github.io/spo.github.io/">Project Page</a></p><p></p><h2 id="abstract-xssdjytwy">Abstract</h2><p>Recently, Direct Preference Optimization (DPO) has extended its success from aligning large language models (LLMs) to aligning text-to-image diffusion models with human preferences. Unlike most existing DPO methods that assume all diffusion steps share a consistent preference order with the final generated images, we argue that this assumption neglects step-specific denoising performance and that preference labels should be tailored to each step's contribution.</p><p>To address this limitation, we propose Step-aware Preference Optimization (SPO), a novel post-training approach that independently evaluates and adjusts the denoising performance at each step, using a <em>step-aware preference model</em> and a <em>step-wise resampler</em> to ensure accurate step-aware supervision. Specifically, at each denoising step, we sample a pool of images, find a suitable win-lose pair, and, most importantly, randomly select a single image from the pool to initialize the next denoising step. This step-wise resampler process ensures the next win-lose image pair comes from the same image, making the win-lose comparison independent of the previous step. To assess the preferences at each step, we train a separate step-aware preference model that can be applied to both noisy and clean images.</p><p>Our experiments with Stable Diffusion v1.5 and SDXL demonstrate that SPO significantly outperforms the latest Diffusion-DPO in aligning generated images with complex, detailed prompts and enhancing aesthetics, while also achieving more than 20× times faster in training efficiency. Code and model: <a target="_blank" rel="ugc" href="https://rockeycoss.github.io/spo.github.io/"><strong><u>https://rockeycoss.github.io/spo.github.io/</u></strong></a></p><p></p><h2 id="model-description-hiz7748c9">Model Description</h2><p>This model is fine-tuned from <a rel="ugc" href="https://huggingface.co/runwayml/stable-diffusion-v1-5">runwayml/stable-diffusion-v1-5</a>. It has been trained on 4,000 prompts for 10 epochs. This checkpoint is a LoRA checkpoint. For more information, please visit <a target="_blank" rel="ugc" href="https://huggingface.co/SPO-Diffusion-Models/SPO-SD-v1-5_4k-p_10ep_LoRA">here</a></p><p></p><h2 id="citation-7xrj1unei">Citation</h2><p>If you find our work useful, please consider giving us a star and citing our work.</p><pre><code>@article{liang2024step,
    title={Step-aware Preference Optimization: Aligning Preference with Denoising Performance at Each Step},
    author={Liang, Zhanhao and Yuan, Yuhui and Gu, Shuyang and Chen, Bohan and Hang, Tiankai and Li, Ji and Zheng, Liang},
    journal={arXiv preprint arXiv:2406.04314},
    year={2024}
  }</code></pre>
modelVersions.availability: Public
modelVersions.baseModel: SD 1.5
modelVersions.downloadUrl: https://civitai.com/api/download/models/584853
modelVersions.downloaded: true
modelVersions.downloadUrl: https://civitai.com/api/download/models/584853
modelVersions.AutoV1: 091C7712
modelVersions.AutoV2: D35E53074E
modelVersions.AutoV3: 8EE3A1D108F9
modelVersions.BLAKE3: 10EFFB1732A9CF064DB3B181E32637B8ABFF41D58DC25627D2569407D7711247
modelVersions.CRC32: FAE401F1
modelVersions.SHA256: D35E53074E19023DB35AB4A58027B7612FE4F079DD03B41C95DFF32722BD2890
modelVersions.format: SafeTensor
modelVersions.name: spo-sd-v1-5_4k-p_10ep_lora_webui.safetensors
modelVersions.pickleScanMessage: No Pickle imports
modelVersions.pickleScanResult: Success
modelVersions.scannedAt: 2024-06-20T02:16:14.840Z
modelVersions.type: Model
modelVersions.virusScanResult: Success
modelVersions.hash: UTG[g4?wJCwb9bERNxxZxvE1RjtRx]%1xFR-
modelVersions.type: image
modelVersions.url: https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/dfbc008d-67e6-45c0-b2a2-d295d9233900/width=450/16425265.jpeg
modelVersions.hash: UbG+8l^,t6jY_4x]NGR*kWxuaes:t6WCRjWB
modelVersions.type: image
modelVersions.url: https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/823d65b2-a195-47c8-b74e-b2b71edbe38a/width=450/16425262.jpeg
modelVersions.hash: UcGb;zO@I9xB%7XORhaKxWRiRPWT%goIr=R+
modelVersions.type: image
modelVersions.url: https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/783ced04-4e94-422a-afd0-14eb10c5aebe/width=450/16425263.jpeg
modelVersions.hash: UMIEhR^+004n4.ofIURP~WRjM{xa%MxuWBRj
modelVersions.type: image
modelVersions.url: https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/54fbec63-bb7b-4bae-93aa-56ca3a9d7c1d/width=450/16425264.jpeg
modelVersions.name: v1.0
modelVersions.publishedAt: 2024-06-20T02:16:26.510Z
name: SPO-SD-v1-5_4k-p_10ep_LoRA_webui
tags: base model