title: Contrastive Prefence Learning: Learning from Human Feedback without RL
summary: Reinforcement Learning from Human Feedback (RLHF) has emerged as a popular
  paradigm for aligning models with human intent. Typically RLHF algorithms
  operate in two phases: first, use human preferences to learn a reward function
  and second, align the model by optimizing the learned reward via reinforcement
  learning (RL). This paradigm assumes that human preferences are distributed
  according to reward, but recent work suggests that they instead follow the
  regret under the user's optimal policy. Thus, learning a reward function from
  feedback is not only based on a flawed assumption of human preference, but also
  leads to unwieldy optimization challenges that stem from policy gradients or
  bootstrapping in the RL phase. Because of these optimization challenges,
  contemporary RLHF methods restrict themselves to contextual bandit settings
  (e.g., as in large language models) or limit observation dimensionality (e.g.,
  state-based robotics). We overcome these limitations by introducing a new
  family of algorithms for optimizing behavior from human feedback using the
  regret-based model of human preferences. Using the principle of maximum
  entropy, we derive Contrastive Preference Learning (CPL), an algorithm for
  learning optimal policies from preferences without learning reward functions,
  circumventing the need for RL. CPL is fully off-policy, uses only a simple
  contrastive objective, and can be applied to arbitrary MDPs. This enables CPL
  to elegantly scale to high-dimensional and sequential RLHF problems while being
  simpler than prior methods.
paper_authors._id: 6535f1a125a58aec65c6ac6f
paper_authors.name: Joey Hejna
paper_authors._id: 6535f1a125a58aec65c6ac70
paper_authors.name: Rafael Rafailov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T08:43:41.615Z
paper_authors._id: 64701c98d742e9ef6522598f
paper_authors.avatarUrl: /avatars/df3a52bd4bc374d7cb4061f449f75c07.svg
paper_authors.fullname: Rafael Rafailov
paper_authors.type: user
paper_authors.user: rmrafailov
paper_authors._id: 6535f1a125a58aec65c6ac71
paper_authors.name: Harshit Sikchi
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-10-23T04:22:55.676Z
paper_authors._id: 6304559b1dd5d3c6248878f1
paper_authors.avatarUrl: /avatars/50ad7ffcc226c9ed7c6a78634a5fc8b4.svg
paper_authors.fullname: Harshit Sikchi
paper_authors.type: user
paper_authors.user: hsikchi
paper_authors._id: 6535f1a125a58aec65c6ac72
paper_authors.name: Chelsea Finn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T08:43:48.764Z
paper_authors._id: 64ac22a9193f0a807deb673d
paper_authors.avatarUrl: /avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
paper_authors.fullname: Chelsea Finn
paper_authors.type: user
paper_authors.user: cbfinn
paper_authors._id: 6535f1a125a58aec65c6ac73
paper_authors.name: Scott Niekum
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-23T14:46:38.498Z
paper_authors._id: 653675a67f86289757f461ca
paper_authors.avatarUrl: /avatars/47fe2c67305a3062c29c27aa2ae642b2.svg
paper_authors.fullname: Scott Niekum
paper_authors.type: user
paper_authors.user: sniekum
paper_authors._id: 6535f1a125a58aec65c6ac74
paper_authors.name: W. Bradley Knox
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-24T07:29:47.953Z
paper_authors._id: 64aee90a710608ee1a01b974
paper_authors.avatarUrl: /avatars/7ea57ca3a4c75ad714a8b5d3f712c4c6.svg
paper_authors.fullname: Brad Knox
paper_authors.type: user
paper_authors.user: wbknox
paper_authors._id: 6535f1a125a58aec65c6ac75
paper_authors.name: Dorsa Sadigh
paper_title: Contrastive Prefence Learning: Learning from Human Feedback without RL
paper_summary: Reinforcement Learning from Human Feedback (RLHF) has emerged as a popular
  paradigm for aligning models with human intent. Typically RLHF algorithms
  operate in two phases: first, use human preferences to learn a reward function
  and second, align the model by optimizing the learned reward via reinforcement
  learning (RL). This paradigm assumes that human preferences are distributed
  according to reward, but recent work suggests that they instead follow the
  regret under the user's optimal policy. Thus, learning a reward function from
  feedback is not only based on a flawed assumption of human preference, but also
  leads to unwieldy optimization challenges that stem from policy gradients or
  bootstrapping in the RL phase. Because of these optimization challenges,
  contemporary RLHF methods restrict themselves to contextual bandit settings
  (e.g., as in large language models) or limit observation dimensionality (e.g.,
  state-based robotics). We overcome these limitations by introducing a new
  family of algorithms for optimizing behavior from human feedback using the
  regret-based model of human preferences. Using the principle of maximum
  entropy, we derive Contrastive Preference Learning (CPL), an algorithm for
  learning optimal policies from preferences without learning reward functions,
  circumventing the need for RL. CPL is fully off-policy, uses only a simple
  contrastive objective, and can be applied to arbitrary MDPs. This enables CPL
  to elegantly scale to high-dimensional and sequential RLHF problems while being
  simpler than prior methods.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) reward function human preferences regret under the user's optimal policy policy gradients bootstrapping contextual bandit settings state-based robotics regret-based model of human preferences maximum entropy Contrastive Preference Learning (CPL) optimal policies contrastive objective MDPs high-dimensional sequential RLHF problems