title: BPO: Supercharging Online Preference Learning by Adhering to the
    Proximity of Behavior LLM
summary: Direct alignment from preferences (DAP) has emerged as a promising paradigm
  for aligning large language models (LLMs) to human desiderata from
  pre-collected, offline preference datasets. While recent studies indicate that
  existing offline DAP methods can directly benefit from online training samples,
  we highlight the need to develop specific online DAP algorithms to fully
  harness the power of online training. Specifically, we identify that the
  learned LLM should adhere to the proximity of the behavior LLM, which collects
  the training samples. To this end, we propose online Preference Optimization in
  proximity to the Behavior LLM (BPO), emphasizing the importance of constructing
  a proper trust region for LLM alignment.
    We conduct extensive experiments to validate the effectiveness and
  applicability of our approach by integrating it with various DAP methods,
  resulting in significant performance improvements across a wide range of tasks
  when training with the same amount of preference data. Even when only
  introducing one additional data collection phase, our online BPO improves its
  offline DAP baseline from 72.0% to 80.2% on TL;DR and from 82.2% to 89.1% on
  Anthropic Helpfulness in terms of win rate against human reference text.
paper_authors._id: 667266a20f6ec76f334bf263
paper_authors.name: Wenda Xu
paper_authors._id: 667266a20f6ec76f334bf264
paper_authors.name: Jiachen Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T07:37:31.704Z
paper_authors._id: 640d7ec5fdeaae13907fc488
paper_authors.avatarUrl: /avatars/acf43ea155105a51c8612dacc4725091.svg
paper_authors.fullname: Jiachen Li
paper_authors.type: user
paper_authors.user: jiachenli-ucsb
paper_authors._id: 667266a20f6ec76f334bf265
paper_authors.name: William Yang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T10:19:15.810Z
paper_authors._id: 6476607efb22e3b77f3f1452
paper_authors.avatarUrl: /avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
paper_authors.fullname: William Yang Wang
paper_authors.type: user
paper_authors.user: wangwilliamyang
paper_authors._id: 667266a20f6ec76f334bf266
paper_authors.name: Lei Li
paper_title: BPO: Supercharging Online Preference Learning by Adhering to the
    Proximity of Behavior LLM
paper_summary: Direct alignment from preferences (DAP) has emerged as a promising paradigm
  for aligning large language models (LLMs) to human desiderata from
  pre-collected, offline preference datasets. While recent studies indicate that
  existing offline DAP methods can directly benefit from online training samples,
  we highlight the need to develop specific online DAP algorithms to fully
  harness the power of online training. Specifically, we identify that the
  learned LLM should adhere to the proximity of the behavior LLM, which collects
  the training samples. To this end, we propose online Preference Optimization in
  proximity to the Behavior LLM (BPO), emphasizing the importance of constructing
  a proper trust region for LLM alignment.
    We conduct extensive experiments to validate the effectiveness and
  applicability of our approach by integrating it with various DAP methods,
  resulting in significant performance improvements across a wide range of tasks
  when training with the same amount of preference data. Even when only
  introducing one additional data collection phase, our online BPO improves its
  offline DAP baseline from 72.0% to 80.2% on TL;DR and from 82.2% to 89.1% on
  Anthropic Helpfulness in terms of win rate against human reference text.
paper_ai_summary: Online Preference Optimization in proximity to the Behavior LLM (BPO) enhances the alignment of large language models to human preferences by integrating offline DAP methods with online training samples, leading to significant performance improvements.
paper_ai_keywords: Direct alignment from preferences DAP large language models LLMs offline preference datasets online DAP algorithms learned LLM behavior LLM trust region preference optimization TL;DR Anthropic Helpfulness win rate