title: Self-Improving Robust Preference Optimization
summary: Both online and offline RLHF methods such as PPO and DPO have been extremely
  successful in aligning AI with human preferences. Despite their success, the
  existing methods suffer from a fundamental problem that their optimal solution
  is highly task-dependent (i.e., not robust to out-of-distribution (OOD) tasks).
  Here we address this challenge by proposing Self-Improving Robust Preference
  Optimization SRPO, a practical and mathematically principled offline RLHF
  framework that is completely robust to the changes in the task. The key idea of
  SRPO is to cast the problem of learning from human preferences as a
  self-improvement process, which can be mathematically expressed in terms of a
  min-max objective that aims at joint optimization of self-improvement policy
  and the generative policy in an adversarial fashion. The solution for this
  optimization problem is independent of the training task and thus it is robust
  to its changes. We then show that this objective can be re-expressed in the
  form of a non-adversarial offline loss which can be optimized using standard
  supervised optimization techniques at scale without any need for reward model
  and online inference. We show the effectiveness of SRPO in terms of AI Win-Rate
  (WR) against human (GOLD) completions. In particular, when SRPO is evaluated on
  the OOD XSUM dataset, it outperforms the celebrated DPO by a clear margin of
  15% after 5 self-revisions, achieving WR of 90%.
paper_authors._id: 665fe77de659976ababbebaf
paper_authors.name: Eugene Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-05T09:35:20.224Z
paper_authors._id: 60c341c722a9d3ec68102981
paper_authors.avatarUrl: /avatars/bac7c3ce9f12d0c16c63ddd31a21e71f.svg
paper_authors.fullname: Eugene Choi
paper_authors.type: user
paper_authors.user: eugenechoi
paper_authors._id: 665fe77de659976ababbebb0
paper_authors.name: Arash Ahmadian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-05T09:35:26.069Z
paper_authors._id: 661b21f8ea926e8f86729e61
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/661b21f8ea926e8f86729e61/gM59piqaqjEu2IPbDvCbZ.jpeg
paper_authors.fullname: Arash Ahmadian
paper_authors.type: user
paper_authors.user: ArashAhmadian
paper_authors._id: 665fe77de659976ababbebb1
paper_authors.name: Matthieu Geist
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-19T13:09:40.615Z
paper_authors._id: 67bef153eaca0bef78ed5478
paper_authors.avatarUrl: /avatars/ab838c541bd170986370a99670981c2f.svg
paper_authors.fullname: Matthieu Geist
paper_authors.type: user
paper_authors.user: matthieu-geist
paper_authors._id: 665fe77de659976ababbebb2
paper_authors.name: Oilvier Pietquin
paper_authors._id: 665fe77de659976ababbebb3
paper_authors.name: Mohammad Gheshlaghi Azar
paper_title: Self-Improving Robust Preference Optimization
paper_summary: Both online and offline RLHF methods such as PPO and DPO have been extremely
  successful in aligning AI with human preferences. Despite their success, the
  existing methods suffer from a fundamental problem that their optimal solution
  is highly task-dependent (i.e., not robust to out-of-distribution (OOD) tasks).
  Here we address this challenge by proposing Self-Improving Robust Preference
  Optimization SRPO, a practical and mathematically principled offline RLHF
  framework that is completely robust to the changes in the task. The key idea of
  SRPO is to cast the problem of learning from human preferences as a
  self-improvement process, which can be mathematically expressed in terms of a
  min-max objective that aims at joint optimization of self-improvement policy
  and the generative policy in an adversarial fashion. The solution for this
  optimization problem is independent of the training task and thus it is robust
  to its changes. We then show that this objective can be re-expressed in the
  form of a non-adversarial offline loss which can be optimized using standard
  supervised optimization techniques at scale without any need for reward model
  and online inference. We show the effectiveness of SRPO in terms of AI Win-Rate
  (WR) against human (GOLD) completions. In particular, when SRPO is evaluated on
  the OOD XSUM dataset, it outperforms the celebrated DPO by a clear margin of
  15% after 5 self-revisions, achieving WR of 90%.
paper_ai_summary: SRPO, a self-improving offline RLHF framework, achieves robustness to out-of-distribution tasks by optimizing a min-max objective that jointly enhances self-improvement and generative policies, leading to superior performance compared to DPO.
paper_ai_keywords: PPO DPO Self-Improving Robust Preference Optimization SRPO offline RLHF self-improvement process min-max objective self-improvement policy generative policy adversarial fashion non-adversarial offline loss supervised optimization reward model