title: Dataset Reset Policy Optimization for RLHF
summary: Reinforcement Learning (RL) from Human Preference-based feedback is a popular
  paradigm for fine-tuning generative models, which has produced impressive
  models such as GPT-4 and Claude3 Opus. This framework often consists of two
  steps: learning a reward model from an offline preference dataset followed by
  running online RL to optimize the learned reward model. In this work,
  leveraging the idea of reset, we propose a new RLHF algorithm with provable
  guarantees. Motivated by the fact that offline preference dataset provides
  informative states (i.e., data that is preferred by the labelers), our new
  algorithm, Dataset Reset Policy Optimization (DR-PO), integrates the existing
  offline preference dataset into the online policy training procedure via
  dataset reset: it directly resets the policy optimizer to the states in the
  offline dataset, instead of always starting from the initial state
  distribution. In theory, we show that DR-PO learns to perform at least as good
  as any policy that is covered by the offline dataset under general function
  approximation with finite sample complexity. In experiments, we demonstrate
  that on both the TL;DR summarization and the Anthropic Helpful Harmful (HH)
  dataset, the generation from DR-PO is better than that from Proximal Policy
  Optimization (PPO) and Direction Preference Optimization (DPO), under the
  metric of GPT4 win-rate. Code for this work can be found at
  https://github.com/Cornell-RL/drpo.
paper_authors._id: 661c8966c42687695eba6e2d
paper_authors.name: Jonathan D. Chang
paper_authors._id: 661c8966c42687695eba6e2e
paper_authors.name: Wenhao Shan
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-05-01T03:45:42.291Z
paper_authors._id: 6631baa4a8374cabcef53de1
paper_authors.avatarUrl: /avatars/ab4f94cdd4104694d170646ed7463fa6.svg
paper_authors.fullname: Wenhao Zhan
paper_authors.type: user
paper_authors.user: whzhan
paper_authors._id: 661c8966c42687695eba6e2f
paper_authors.name: Owen Oertell
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-16T00:02:57.340Z
paper_authors._id: 65786ad2821f439498e5d584
paper_authors.avatarUrl: /avatars/1a85ba5b7f6c6df32d12397057b04566.svg
paper_authors.fullname: Owen Oertell
paper_authors.type: user
paper_authors.user: ojo2
paper_authors._id: 661c8966c42687695eba6e30
paper_authors.name: Kianté Brantley
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-19T07:39:06.629Z
paper_authors._id: 644eb204722da4dfd871f1fa
paper_authors.avatarUrl: /avatars/e09550b70161f8e541b91a019a4702b9.svg
paper_authors.fullname: kiante
paper_authors.type: user
paper_authors.user: xkianteb
paper_authors._id: 661c8966c42687695eba6e31
paper_authors.name: Dipendra Misra
paper_authors._id: 661c8966c42687695eba6e32
paper_authors.name: Jason D. Lee
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-04-15T01:56:55.265Z
paper_authors._id: 64ad70f7f9f761a4c3910c63
paper_authors.avatarUrl: /avatars/69b6c167299ca7a174a69ddc5a95e4a2.svg
paper_authors.fullname: Jason Lee
paper_authors.type: user
paper_authors.user: Jasondlee
paper_authors._id: 661c8966c42687695eba6e33
paper_authors.name: Wen Sun
paper_title: Dataset Reset Policy Optimization for RLHF
paper_summary: Reinforcement Learning (RL) from Human Preference-based feedback is a popular
  paradigm for fine-tuning generative models, which has produced impressive
  models such as GPT-4 and Claude3 Opus. This framework often consists of two
  steps: learning a reward model from an offline preference dataset followed by
  running online RL to optimize the learned reward model. In this work,
  leveraging the idea of reset, we propose a new RLHF algorithm with provable
  guarantees. Motivated by the fact that offline preference dataset provides
  informative states (i.e., data that is preferred by the labelers), our new
  algorithm, Dataset Reset Policy Optimization (DR-PO), integrates the existing
  offline preference dataset into the online policy training procedure via
  dataset reset: it directly resets the policy optimizer to the states in the
  offline dataset, instead of always starting from the initial state
  distribution. In theory, we show that DR-PO learns to perform at least as good
  as any policy that is covered by the offline dataset under general function
  approximation with finite sample complexity. In experiments, we demonstrate
  that on both the TL;DR summarization and the Anthropic Helpful Harmful (HH)
  dataset, the generation from DR-PO is better than that from Proximal Policy
  Optimization (PPO) and Direction Preference Optimization (DPO), under the
  metric of GPT4 win-rate. Code for this work can be found at
  https://github.com/Cornell-RL/drpo.
paper_ai_keywords: reinforcement learning from human preference (RLHF) generative models reward model offline preference dataset online RL reset Dataset Reset Policy Optimization (DR-PO) policy optimizer function approximation finite sample complexity TL;DR summarization Anthropic Helpful Harmful (HH) dataset Proximal Policy Optimization (PPO) Direction Preference Optimization (DPO) GPT4 win-rate