description: <h1 id="step-aware-preference-optimization:-aligning-preference-with-denoising-performance-at-each-step-ys94rv4y1">Step-aware Preference Optimization: Aligning Preference with Denoising Performance at Each Step</h1><p></p><p><a target="_blank" rel="ugc" href="https://arxiv.org/abs/2406.04314">Arxiv Paper</a></p><p><a target="_blank" rel="ugc" href="https://github.com/RockeyCoss/SPO">Github Code</a></p><p><a target="_blank" rel="ugc" href="https://rockeycoss.github.io/spo.github.io/">Project Page</a></p><p></p><h2 id="abstract-fmv0mryvo">Abstract</h2><p>Recently, Direct Preference Optimization (DPO) has extended its success from aligning large language models (LLMs) to aligning text-to-image diffusion models with human preferences. Unlike most existing DPO methods that assume all diffusion steps share a consistent preference order with the final generated images, we argue that this assumption neglects step-specific denoising performance and that preference labels should be tailored to each step's contribution.</p><p>To address this limitation, we propose Step-aware Preference Optimization (SPO), a novel post-training approach that independently evaluates and adjusts the denoising performance at each step, using a <em>step-aware preference model</em> and a <em>step-wise resampler</em> to ensure accurate step-aware supervision. Specifically, at each denoising step, we sample a pool of images, find a suitable win-lose pair, and, most importantly, randomly select a single image from the pool to initialize the next denoising step. This step-wise resampler process ensures the next win-lose image pair comes from the same image, making the win-lose comparison independent of the previous step. To assess the preferences at each step, we train a separate step-aware preference model that can be applied to both noisy and clean images.</p><p>Our experiments with Stable Diffusion v1.5 and SDXL demonstrate that SPO significantly outperforms the latest Diffusion-DPO in aligning generated images with complex, detailed prompts and enhancing aesthetics, while also achieving more than 20× times faster in training efficiency. Code and model: <a target="_blank" rel="ugc" href="https://rockeycoss.github.io/spo.github.io/"><strong><u>https://rockeycoss.github.io/spo.github.io/</u></strong></a></p><p></p><h2 id="model-description-uw2dzpmby">Model Description</h2><p>This model is fine-tuned from <a target="_blank" rel="ugc" href="https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0"><strong><u>stable-diffusion-xl-base-1.0</u></strong></a>. It has been trained on 4,000 prompts for 10 epochs. This checkpoint is a LoRA checkpoint. For more information, please visit <a target="_blank" rel="ugc" href="https://huggingface.co/SPO-Diffusion-Models/SPO-SDXL_4k-p_10ep_LoRA">here</a></p><p></p><h2 id="citation-zu2welgf5">Citation</h2><p>If you find our work useful, please consider giving us a star and citing our work.</p><pre><code>@article{liang2024step,
    title={Step-aware Preference Optimization: Aligning Preference with Denoising Performance at Each Step},
    author={Liang, Zhanhao and Yuan, Yuhui and Gu, Shuyang and Chen, Bohan and Hang, Tiankai and Li, Ji and Zheng, Liang},
    journal={arXiv preprint arXiv:2406.04314},
    year={2024}
  }</code></pre>
modelVersions.availability: Public
modelVersions.baseModel: SDXL 1.0
modelVersions.downloadUrl: https://civitai.com/api/download/models/567119
modelVersions.downloaded: true
modelVersions.downloadUrl: https://civitai.com/api/download/models/567119
modelVersions.AutoV1: 6ED9C1EB
modelVersions.AutoV2: B6C2C16F3E
modelVersions.AutoV3: 5256EE0C312F
modelVersions.BLAKE3: 77A12904B635016D196C2546A1522ADF0E0F2C82EB4777AA3FC20B79889F454C
modelVersions.CRC32: 620AF0C8
modelVersions.SHA256: B6C2C16F3EF579885F10E94468D8F7196D09464002D116C115432207F4B1F8AB
modelVersions.format: SafeTensor
modelVersions.name: spo_sdxl_10ep_4k-data_lora_webui.safetensors
modelVersions.pickleScanMessage: No Pickle imports
modelVersions.pickleScanResult: Success
modelVersions.scannedAt: 2024-06-12T01:25:56.123Z
modelVersions.type: Model
modelVersions.virusScanResult: Success
modelVersions.hash: UBH_6^I;rqkXS~tSR*xt?a~WS5waA0%Li_RO
modelVersions.type: image
modelVersions.url: https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/da975e99-fed9-4d54-a79c-524c89db1724/width=450/15471700.jpeg
modelVersions.hash: UDG87sM|oO~C?aNGT0RjEOs9k99Z5Q%2-TRk
modelVersions.type: image
modelVersions.url: https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/96015beb-c363-4779-87ee-6aa92664f840/width=450/15471753.jpeg
modelVersions.hash: UBI#V}t7?vtQ~Vj]?aj@9tWBnPof_3WV^+t6
modelVersions.type: image
modelVersions.url: https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/53a1123b-5018-491f-9462-751f0f8bc078/width=450/15471752.jpeg
modelVersions.hash: UNIX:C01smae~qM{%2of9v%1%1oJt7bHRka}
modelVersions.type: image
modelVersions.url: https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/49edbfab-026f-4a29-9925-32cf023558f2/width=450/15471751.jpeg
modelVersions.name: v1.0
modelVersions.publishedAt: 2024-06-12T01:32:08.671Z
name: SPO-SDXL_4k-p_10ep_LoRA_webui
tags: base model