PILAF: Optimal Human Preference Sampling for Reward Modeling
As large language models increasingly drive real-world applications, aligning
them with human values becomes paramount. Reinforcement Learning from Human
Feedback (RLHF) has emerged as a key technique, translating preference data
into reward models when oracle human values remain inaccessible. In practice,
RLHF mostly relies on approximate reward models, which may not consistently
guide the policy toward maximizing the underlying human values. We propose
Policy-Interpolated Learning for Aligned Feedback (PILAF), a novel response
sampling strategy for preference labeling that explicitly aligns preference
learning with maximizing the underlying oracle reward. PILAF is theoretically
grounded, demonstrating optimality from both an optimization and a statistical
perspective. The method is straightforward to implement and demonstrates strong
performance in iterative and online RLHF settings where feedback curation is
critical.
67a5882fa8e877ef10b8d1fd
Yunzhen Feng
admin_assigned
2025-02-07T10:49:26.187Z
664187fa1cd689758847f44b
/avatars/501ed1d5bcffd7466fd8b8c8d3b758f0.svg
Yunzhen Feng
user
Coolfyz
67a5882fa8e877ef10b8d1fe
Ariel Kwiatkowski
admin_assigned
2025-02-07T10:49:33.321Z
625de0717341c641426e7932
/avatars/9deb06fc565a80002c3ae75c6f4cd9e7.svg
Ariel Kwiatkowski
user
RedTachyon
67a5882fa8e877ef10b8d1ff
Kunhao Zheng
admin_assigned
2025-02-07T10:49:41.052Z
6424123d3fa01ecba6fd94e8
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/B-1YSkTJMVBBQDX3WVxIL.jpeg
Kunhao Zheng
user
Kunhao
67a5882fa8e877ef10b8d200
Julia Kempe
admin_assigned
2025-02-07T10:49:48.009Z
65ce30e06da01df536eded5a
/avatars/04c32cba7a3bbaf9ea5dee88c96cf87b.svg
Julia Kempe
user
Knykny
67a5882fa8e877ef10b8d201
Yaqi Duan
admin_assigned
2025-02-07T10:49:54.220Z
66494b428d50b4b0efceab9c
/avatars/ac7293aafaf15759d53cf62f4e1ae874.svg
Yaqi Duan
user
duanyq
PILAF: Optimal Human Preference Sampling for Reward Modeling
As large language models increasingly drive real-world applications, aligning
them with human values becomes paramount. Reinforcement Learning from Human
Feedback (RLHF) has emerged as a key technique, translating preference data
into reward models when oracle human values remain inaccessible. In practice,
RLHF mostly relies on approximate reward models, which may not consistently
guide the policy toward maximizing the underlying human values. We propose
Policy-Interpolated Learning for Aligned Feedback (PILAF), a novel response
sampling strategy for preference labeling that explicitly aligns preference
learning with maximizing the underlying oracle reward. PILAF is theoretically
grounded, demonstrating optimality from both an optimization and a statistical
perspective. The method is straightforward to implement and demonstrates strong
performance in iterative and online RLHF settings where feedback curation is
critical.
PILAF is a novel response sampling strategy in RLHF that aligns preference learning with human values, showing strong performance in both iterative and online settings.
Reinforcement Learning from Human Feedback RLHF approximate reward models Policy-Interpolated Learning for Aligned Feedback PILAF preference learning oracle reward