title: Understanding the performance gap between online and offline alignment
    algorithms
summary: Reinforcement learning from human feedback (RLHF) is the canonical framework
  for large language model alignment. However, rising popularity in offline
  alignment algorithms challenge the need for on-policy sampling in RLHF. Within
  the context of reward over-optimization, we start with an opening set of
  experiments that demonstrate the clear advantage of online methods over offline
  methods. This prompts us to investigate the causes to the performance
  discrepancy through a series of carefully designed experimental ablations. We
  show empirically that hypotheses such as offline data coverage and data quality
  by itself cannot convincingly explain the performance difference. We also find
  that while offline algorithms train policy to become good at pairwise
  classification, it is worse at generations; in the meantime the policies
  trained by online algorithms are good at generations while worse at pairwise
  classification. This hints at a unique interplay between discriminative and
  generative capabilities, which is greatly impacted by the sampling process.
  Lastly, we observe that the performance discrepancy persists for both
  contrastive and non-contrastive loss functions, and appears not to be addressed
  by simply scaling up policy networks. Taken together, our study sheds light on
  the pivotal role of on-policy sampling in AI alignment, and hints at certain
  fundamental challenges of offline alignment algorithms.
paper_authors._id: 6644fa49123d31ba5b288503
paper_authors.name: Yunhao Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-15T20:15:43.375Z
paper_authors._id: 62ac59847bd21e09d0053375
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1655462260649-noauth.jpeg
paper_authors.fullname: Robin Tang
paper_authors.type: user
paper_authors.user: LemonPlay
paper_authors._id: 6644fa49123d31ba5b288504
paper_authors.name: Daniel Zhaohan Guo
paper_authors._id: 6644fa49123d31ba5b288505
paper_authors.name: Zeyu Zheng
paper_authors._id: 6644fa49123d31ba5b288506
paper_authors.name: Daniele Calandriello
paper_authors._id: 6644fa49123d31ba5b288507
paper_authors.name: Yuan Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-15T20:14:18.573Z
paper_authors._id: 6433a8c37b8247480105cf7f
paper_authors.avatarUrl: /avatars/a19536f3f3b8eae3609648bea112b381.svg
paper_authors.fullname: Yuan Cao
paper_authors.type: user
paper_authors.user: caoyuan33
paper_authors._id: 6644fa49123d31ba5b288508
paper_authors.name: Eugene Tarassov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T14:24:09.016Z
paper_authors._id: 63280235bfc72a1d59d029c3
paper_authors.avatarUrl: /avatars/85be03ddbe48a0d7a12f8fd95eeb00a1.svg
paper_authors.fullname: Eugene Tarassov
paper_authors.type: user
paper_authors.user: etarassov
paper_authors._id: 6644fa49123d31ba5b288509
paper_authors.name: Rémi Munos
paper_authors._id: 6644fa49123d31ba5b28850a
paper_authors.name: Bernardo Ávila Pires
paper_authors._id: 6644fa49123d31ba5b28850b
paper_authors.name: Michal Valko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-15T20:14:50.193Z
paper_authors._id: 651e97156d92456bdf5ace6b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/651e97156d92456bdf5ace6b/QDQN6LXnCkyjRQAp86fwE.jpeg
paper_authors.fullname: Michal Valko
paper_authors.type: user
paper_authors.user: misovalko
paper_authors._id: 6644fa49123d31ba5b28850c
paper_authors.name: Yong Cheng
paper_authors._id: 6644fa49123d31ba5b28850d
paper_authors.name: Will Dabney
paper_title: Understanding the performance gap between online and offline alignment
    algorithms
paper_summary: Reinforcement learning from human feedback (RLHF) is the canonical framework
  for large language model alignment. However, rising popularity in offline
  alignment algorithms challenge the need for on-policy sampling in RLHF. Within
  the context of reward over-optimization, we start with an opening set of
  experiments that demonstrate the clear advantage of online methods over offline
  methods. This prompts us to investigate the causes to the performance
  discrepancy through a series of carefully designed experimental ablations. We
  show empirically that hypotheses such as offline data coverage and data quality
  by itself cannot convincingly explain the performance difference. We also find
  that while offline algorithms train policy to become good at pairwise
  classification, it is worse at generations; in the meantime the policies
  trained by online algorithms are good at generations while worse at pairwise
  classification. This hints at a unique interplay between discriminative and
  generative capabilities, which is greatly impacted by the sampling process.
  Lastly, we observe that the performance discrepancy persists for both
  contrastive and non-contrastive loss functions, and appears not to be addressed
  by simply scaling up policy networks. Taken together, our study sheds light on
  the pivotal role of on-policy sampling in AI alignment, and hints at certain
  fundamental challenges of offline alignment algorithms.
paper_ai_keywords: Reinforcement learning from human feedback large language model alignment on-policy sampling reward over-optimization offline methods online methods experimental ablations offline data coverage data quality pairwise classification generative capabilities contrastive loss functions Non-contrastive loss functions policy networks AI alignment