title: PILAF: Optimal Human Preference Sampling for Reward Modeling
summary: As large language models increasingly drive real-world applications, aligning
  them with human values becomes paramount. Reinforcement Learning from Human
  Feedback (RLHF) has emerged as a key technique, translating preference data
  into reward models when oracle human values remain inaccessible. In practice,
  RLHF mostly relies on approximate reward models, which may not consistently
  guide the policy toward maximizing the underlying human values. We propose
  Policy-Interpolated Learning for Aligned Feedback (PILAF), a novel response
  sampling strategy for preference labeling that explicitly aligns preference
  learning with maximizing the underlying oracle reward. PILAF is theoretically
  grounded, demonstrating optimality from both an optimization and a statistical
  perspective. The method is straightforward to implement and demonstrates strong
  performance in iterative and online RLHF settings where feedback curation is
  critical.
paper_authors._id: 67a5882fa8e877ef10b8d1fd
paper_authors.name: Yunzhen Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:49:26.187Z
paper_authors._id: 664187fa1cd689758847f44b
paper_authors.avatarUrl: /avatars/501ed1d5bcffd7466fd8b8c8d3b758f0.svg
paper_authors.fullname: Yunzhen Feng
paper_authors.type: user
paper_authors.user: Coolfyz
paper_authors._id: 67a5882fa8e877ef10b8d1fe
paper_authors.name: Ariel Kwiatkowski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:49:33.321Z
paper_authors._id: 625de0717341c641426e7932
paper_authors.avatarUrl: /avatars/9deb06fc565a80002c3ae75c6f4cd9e7.svg
paper_authors.fullname: Ariel Kwiatkowski
paper_authors.type: user
paper_authors.user: RedTachyon
paper_authors._id: 67a5882fa8e877ef10b8d1ff
paper_authors.name: Kunhao Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:49:41.052Z
paper_authors._id: 6424123d3fa01ecba6fd94e8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/B-1YSkTJMVBBQDX3WVxIL.jpeg
paper_authors.fullname: Kunhao Zheng
paper_authors.type: user
paper_authors.user: Kunhao
paper_authors._id: 67a5882fa8e877ef10b8d200
paper_authors.name: Julia Kempe
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:49:48.009Z
paper_authors._id: 65ce30e06da01df536eded5a
paper_authors.avatarUrl: /avatars/04c32cba7a3bbaf9ea5dee88c96cf87b.svg
paper_authors.fullname: Julia Kempe
paper_authors.type: user
paper_authors.user: Knykny
paper_authors._id: 67a5882fa8e877ef10b8d201
paper_authors.name: Yaqi Duan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:49:54.220Z
paper_authors._id: 66494b428d50b4b0efceab9c
paper_authors.avatarUrl: /avatars/ac7293aafaf15759d53cf62f4e1ae874.svg
paper_authors.fullname: Yaqi Duan
paper_authors.type: user
paper_authors.user: duanyq
paper_title: PILAF: Optimal Human Preference Sampling for Reward Modeling
paper_summary: As large language models increasingly drive real-world applications, aligning
  them with human values becomes paramount. Reinforcement Learning from Human
  Feedback (RLHF) has emerged as a key technique, translating preference data
  into reward models when oracle human values remain inaccessible. In practice,
  RLHF mostly relies on approximate reward models, which may not consistently
  guide the policy toward maximizing the underlying human values. We propose
  Policy-Interpolated Learning for Aligned Feedback (PILAF), a novel response
  sampling strategy for preference labeling that explicitly aligns preference
  learning with maximizing the underlying oracle reward. PILAF is theoretically
  grounded, demonstrating optimality from both an optimization and a statistical
  perspective. The method is straightforward to implement and demonstrates strong
  performance in iterative and online RLHF settings where feedback curation is
  critical.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) reward models policy oracle reward Policy-Interpolated Learning for Aligned Feedback (PILAF) preference sampling preference learning iterative online RLHF feedback curation