title: Value-Incentivized Preference Optimization: A Unified Approach to Online
    and Offline RLHF
summary: Reinforcement learning from human feedback (RLHF) has demonstrated great
  promise in aligning large language models (LLMs) with human preference.
  Depending on the availability of preference data, both online and offline RLHF
  are active areas of investigation. A key bottleneck is understanding how to
  incorporate uncertainty estimation in the reward function learned from the
  preference data for RLHF, regardless of how the preference data is collected.
  While the principles of optimism or pessimism under uncertainty are
  well-established in standard reinforcement learning (RL), a
  practically-implementable and theoretically-grounded form amenable to large
  language models is not yet available, as standard techniques for constructing
  confidence intervals become intractable under arbitrary policy
  parameterizations.
    In this paper, we introduce a unified approach to online and offline RLHF --
  value-incentivized preference optimization (VPO) -- which regularizes the
  maximum-likelihood estimate of the reward function with the corresponding value
  function, modulated by a sign to indicate whether the optimism or
  pessimism is chosen. VPO also directly optimizes the policy with implicit
  reward modeling, and therefore shares a simpler RLHF pipeline similar to direct
  preference optimization. Theoretical guarantees of VPO are provided for both
  online and offline settings, matching the rates of their standard RL
  counterparts. Moreover, experiments on text summarization and dialog verify the
  practicality and effectiveness of VPO.
paper_authors._id: 6657e4b82387046cf64680cf
paper_authors.name: Shicong Cen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T11:12:54.974Z
paper_authors._id: 65e9731e6434c9e6f4ad7eba
paper_authors.avatarUrl: /avatars/fc1b4996364180cd87e30ad0fbdaf401.svg
paper_authors.fullname: Shicong Cen
paper_authors.type: user
paper_authors.user: tsen9731
paper_authors._id: 6657e4b82387046cf64680d0
paper_authors.name: Jincheng Mei
paper_authors._id: 6657e4b82387046cf64680d1
paper_authors.name: Katayoon Goshvadi
paper_authors._id: 6657e4b82387046cf64680d2
paper_authors.name: Hanjun Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T11:13:10.969Z
paper_authors._id: 642bd29877967049cf4a7877
paper_authors.avatarUrl: /avatars/41ecd2074b152891073a4af1f5cc220a.svg
paper_authors.fullname: Hanjun Dai
paper_authors.type: user
paper_authors.user: kangaroo0123
paper_authors._id: 6657e4b82387046cf64680d3
paper_authors.name: Tong Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T11:13:29.920Z
paper_authors._id: 643d901e37ca9134b5888c51
paper_authors.avatarUrl: /avatars/a2f4a7959b2effed04fb193cee405da7.svg
paper_authors.fullname: Tony Yang
paper_authors.type: user
paper_authors.user: Tongyang
paper_authors._id: 6657e4b82387046cf64680d4
paper_authors.name: Sherry Yang
paper_authors._id: 6657e4b82387046cf64680d5
paper_authors.name: Dale Schuurmans
paper_authors._id: 6657e4b82387046cf64680d6
paper_authors.name: Yuejie Chi
paper_authors._id: 6657e4b82387046cf64680d7
paper_authors.name: Bo Dai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-04T07:32:03.454Z
paper_authors._id: 665e2bd0fbaa279db369b488
paper_authors.avatarUrl: /avatars/8cbc18d17d2813617976dbe709235799.svg
paper_authors.fullname: Bo Dai
paper_authors.type: user
paper_authors.user: doubling
paper_title: Value-Incentivized Preference Optimization: A Unified Approach to Online
    and Offline RLHF
paper_summary: Reinforcement learning from human feedback (RLHF) has demonstrated great
  promise in aligning large language models (LLMs) with human preference.
  Depending on the availability of preference data, both online and offline RLHF
  are active areas of investigation. A key bottleneck is understanding how to
  incorporate uncertainty estimation in the reward function learned from the
  preference data for RLHF, regardless of how the preference data is collected.
  While the principles of optimism or pessimism under uncertainty are
  well-established in standard reinforcement learning (RL), a
  practically-implementable and theoretically-grounded form amenable to large
  language models is not yet available, as standard techniques for constructing
  confidence intervals become intractable under arbitrary policy
  parameterizations.
    In this paper, we introduce a unified approach to online and offline RLHF --
  value-incentivized preference optimization (VPO) -- which regularizes the
  maximum-likelihood estimate of the reward function with the corresponding value
  function, modulated by a sign to indicate whether the optimism or
  pessimism is chosen. VPO also directly optimizes the policy with implicit
  reward modeling, and therefore shares a simpler RLHF pipeline similar to direct
  preference optimization. Theoretical guarantees of VPO are provided for both
  online and offline settings, matching the rates of their standard RL
  counterparts. Moreover, experiments on text summarization and dialog verify the
  practicality and effectiveness of VPO.
paper_ai_keywords: reinforcement learning from human feedback (RLHF) large language models (LLMs) preference data reward function optimism under uncertainty pessimism under uncertainty standard reinforcement learning (RL) confidence intervals policy parameterizations value-incentivized preferences optimization (VPO) maximum-likelihood estimate value function direct preference optimization text summarization dialog