title: SePPO: Semi-Policy Preference Optimization for Diffusion Alignment
summary: Reinforcement learning from human feedback (RLHF) methods are emerging as a
  way to fine-tune diffusion models (DMs) for visual generation. However,
  commonly used on-policy strategies are limited by the generalization capability
  of the reward model, while off-policy approaches require large amounts of
  difficult-to-obtain paired human-annotated data, particularly in visual
  generation tasks. To address the limitations of both on- and off-policy RLHF,
  we propose a preference optimization method that aligns DMs with preferences
  without relying on reward models or paired human-annotated data. Specifically,
  we introduce a Semi-Policy Preference Optimization (SePPO) method. SePPO
  leverages previous checkpoints as reference models while using them to generate
  on-policy reference samples, which replace "losing images" in preference pairs.
  This approach allows us to optimize using only off-policy "winning images."
  Furthermore, we design a strategy for reference model selection that expands
  the exploration in the policy space. Notably, we do not simply treat reference
  samples as negative examples for learning. Instead, we design an anchor-based
  criterion to assess whether the reference samples are likely to be winning or
  losing images, allowing the model to selectively learn from the generated
  reference samples. This approach mitigates performance degradation caused by
  the uncertainty in reference sample quality. We validate SePPO across both
  text-to-image and text-to-video benchmarks. SePPO surpasses all previous
  approaches on the text-to-image benchmarks and also demonstrates outstanding
  performance on the text-to-video benchmarks. Code will be released in
  https://github.com/DwanZhang-AI/SePPO.
paper_authors._id: 670540d5ff94857c894b1595
paper_authors.name: Daoan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T15:37:15.699Z
paper_authors._id: 6241b95cfee9374a2598ecfe
paper_authors.avatarUrl: /avatars/196669df1689a5872fc18b271e80fdc1.svg
paper_authors.fullname: Zhang Daoan
paper_authors.type: user
paper_authors.user: hazard
paper_authors._id: 670540d5ff94857c894b1596
paper_authors.name: Guangchen Lan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-09T07:37:54.329Z
paper_authors._id: 64ff4b1a0e8369f6a8c47c7e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ff4b1a0e8369f6a8c47c7e/X8ocOagj89gexSa253q_8.png
paper_authors.fullname: Eric Lan
paper_authors.type: user
paper_authors.user: Eric-Lan
paper_authors._id: 670540d5ff94857c894b1597
paper_authors.name: Dong-Jun Han
paper_authors._id: 670540d5ff94857c894b1598
paper_authors.name: Wenlin Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T15:37:40.892Z
paper_authors._id: 634f18e4aae4bde2c8e2adca
paper_authors.avatarUrl: /avatars/40549a59fc5ba04a4baa5a1d5dba0847.svg
paper_authors.fullname: Wenlin Yao
paper_authors.type: user
paper_authors.user: wenlinyao
paper_authors._id: 670540d5ff94857c894b1599
paper_authors.name: Xiaoman Pan
paper_authors._id: 670540d5ff94857c894b159a
paper_authors.name: Hongming Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T15:38:12.792Z
paper_authors._id: 64ed478bec06efeb03034933
paper_authors.avatarUrl: /avatars/cd7dc3165831e90cb36d39d41c3c8157.svg
paper_authors.fullname: Hongming Zhang
paper_authors.type: user
paper_authors.user: Hongming98
paper_authors._id: 670540d5ff94857c894b159b
paper_authors.name: Mingxiao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T15:38:21.097Z
paper_authors._id: 63457bb1abd499c80af15dcc
paper_authors.avatarUrl: /avatars/dddfdc51a50f48d97c4308b2c169433c.svg
paper_authors.fullname: MingxiaoLi
paper_authors.type: user
paper_authors.user: Lixiaoming
paper_authors._id: 670540d5ff94857c894b159c
paper_authors.name: Pengcheng Chen
paper_authors._id: 670540d5ff94857c894b159d
paper_authors.name: Yu Dong
paper_authors._id: 670540d5ff94857c894b159e
paper_authors.name: Christopher Brinton
paper_authors._id: 670540d5ff94857c894b159f
paper_authors.name: Jiebo Luo
paper_title: SePPO: Semi-Policy Preference Optimization for Diffusion Alignment
paper_summary: Reinforcement learning from human feedback (RLHF) methods are emerging as a
  way to fine-tune diffusion models (DMs) for visual generation. However,
  commonly used on-policy strategies are limited by the generalization capability
  of the reward model, while off-policy approaches require large amounts of
  difficult-to-obtain paired human-annotated data, particularly in visual
  generation tasks. To address the limitations of both on- and off-policy RLHF,
  we propose a preference optimization method that aligns DMs with preferences
  without relying on reward models or paired human-annotated data. Specifically,
  we introduce a Semi-Policy Preference Optimization (SePPO) method. SePPO
  leverages previous checkpoints as reference models while using them to generate
  on-policy reference samples, which replace "losing images" in preference pairs.
  This approach allows us to optimize using only off-policy "winning images."
  Furthermore, we design a strategy for reference model selection that expands
  the exploration in the policy space. Notably, we do not simply treat reference
  samples as negative examples for learning. Instead, we design an anchor-based
  criterion to assess whether the reference samples are likely to be winning or
  losing images, allowing the model to selectively learn from the generated
  reference samples. This approach mitigates performance degradation caused by
  the uncertainty in reference sample quality. We validate SePPO across both
  text-to-image and text-to-video benchmarks. SePPO surpasses all previous
  approaches on the text-to-image benchmarks and also demonstrates outstanding
  performance on the text-to-video benchmarks. Code will be released in
  https://github.com/DwanZhang-AI/SePPO.
paper_ai_keywords: FederRLHF diffusion models visual generation on-policy strategies off-policy approaches reward model preference optimization preference pairs Semi-Policy Preference Optimization (SePPO) reference models on-policy reference samples off-policy "winning images" reference sample selection anchor-based criterion performance degradation uncertainty in reference sample quality text-to-image benchmarks text-to-video benchmarks