Value-Incentivized Preference Optimization: A Unified Approach to Online
  and Offline RLHF
Reinforcement learning from human feedback (RLHF) has demonstrated great
promise in aligning large language models (LLMs) with human preference.
Depending on the availability of preference data, both online and offline RLHF
are active areas of investigation. A key bottleneck is understanding how to
incorporate uncertainty estimation in the reward function learned from the
preference data for RLHF, regardless of how the preference data is collected.
While the principles of optimism or pessimism under uncertainty are
well-established in standard reinforcement learning (RL), a
practically-implementable and theoretically-grounded form amenable to large
language models is not yet available, as standard techniques for constructing
confidence intervals become intractable under arbitrary policy
parameterizations.
  In this paper, we introduce a unified approach to online and offline RLHF --
value-incentivized preference optimization (VPO) -- which regularizes the
maximum-likelihood estimate of the reward function with the corresponding value
function, modulated by a sign to indicate whether the optimism or
pessimism is chosen. VPO also directly optimizes the policy with implicit
reward modeling, and therefore shares a simpler RLHF pipeline similar to direct
preference optimization. Theoretical guarantees of VPO are provided for both
online and offline settings, matching the rates of their standard RL
counterparts. Moreover, experiments on text summarization and dialog verify the
practicality and effectiveness of VPO.
6657e4b82387046cf64680cf
Shicong Cen
admin_assigned
2024-05-31T11:12:54.974Z
65e9731e6434c9e6f4ad7eba
/avatars/fc1b4996364180cd87e30ad0fbdaf401.svg
Shicong Cen
user
tsen9731
6657e4b82387046cf64680d0
Jincheng Mei
6657e4b82387046cf64680d1
Katayoon Goshvadi
6657e4b82387046cf64680d2
Hanjun Dai
admin_assigned
2024-05-31T11:13:10.969Z
642bd29877967049cf4a7877
/avatars/41ecd2074b152891073a4af1f5cc220a.svg
Hanjun Dai
user
kangaroo0123
6657e4b82387046cf64680d3
Tong Yang
admin_assigned
2024-05-31T11:13:29.920Z
643d901e37ca9134b5888c51
/avatars/a2f4a7959b2effed04fb193cee405da7.svg
Tony Yang
user
Tongyang
6657e4b82387046cf64680d4
Sherry Yang
6657e4b82387046cf64680d5
Dale Schuurmans
6657e4b82387046cf64680d6
Yuejie Chi
6657e4b82387046cf64680d7
Bo Dai
claimed_verified
2024-06-04T07:32:03.454Z
665e2bd0fbaa279db369b488
/avatars/8cbc18d17d2813617976dbe709235799.svg
Bo Dai
user
doubling
Value-Incentivized Preference Optimization: A Unified Approach to Online
  and Offline RLHF
Reinforcement learning from human feedback (RLHF) has demonstrated great
promise in aligning large language models (LLMs) with human preference.
Depending on the availability of preference data, both online and offline RLHF
are active areas of investigation. A key bottleneck is understanding how to
incorporate uncertainty estimation in the reward function learned from the
preference data for RLHF, regardless of how the preference data is collected.
While the principles of optimism or pessimism under uncertainty are
well-established in standard reinforcement learning (RL), a
practically-implementable and theoretically-grounded form amenable to large
language models is not yet available, as standard techniques for constructing
confidence intervals become intractable under arbitrary policy
parameterizations.
  In this paper, we introduce a unified approach to online and offline RLHF --
value-incentivized preference optimization (VPO) -- which regularizes the
maximum-likelihood estimate of the reward function with the corresponding value
function, modulated by a sign to indicate whether the optimism or
pessimism is chosen. VPO also directly optimizes the policy with implicit
reward modeling, and therefore shares a simpler RLHF pipeline similar to direct
preference optimization. Theoretical guarantees of VPO are provided for both
online and offline settings, matching the rates of their standard RL
counterparts. Moreover, experiments on text summarization and dialog verify the
practicality and effectiveness of VPO.
A value-incentivized preference optimization method is introduced to address uncertainty in reinforcement learning from human feedback, offering theoretical guarantees and practical effectiveness in text summarization and dialog.
reinforcement learning from human feedback large language models preference data uncertainty estimation reward function optimism pessimism value-incentivized preference optimization value function direct preference optimization policy optimization theoretical guarantees text summarization dialog