Understanding the performance gap between online and offline alignment
  algorithms
Reinforcement learning from human feedback (RLHF) is the canonical framework
for large language model alignment. However, rising popularity in offline
alignment algorithms challenge the need for on-policy sampling in RLHF. Within
the context of reward over-optimization, we start with an opening set of
experiments that demonstrate the clear advantage of online methods over offline
methods. This prompts us to investigate the causes to the performance
discrepancy through a series of carefully designed experimental ablations. We
show empirically that hypotheses such as offline data coverage and data quality
by itself cannot convincingly explain the performance difference. We also find
that while offline algorithms train policy to become good at pairwise
classification, it is worse at generations; in the meantime the policies
trained by online algorithms are good at generations while worse at pairwise
classification. This hints at a unique interplay between discriminative and
generative capabilities, which is greatly impacted by the sampling process.
Lastly, we observe that the performance discrepancy persists for both
contrastive and non-contrastive loss functions, and appears not to be addressed
by simply scaling up policy networks. Taken together, our study sheds light on
the pivotal role of on-policy sampling in AI alignment, and hints at certain
fundamental challenges of offline alignment algorithms.
6644fa49123d31ba5b288503
Yunhao Tang
admin_assigned
2024-05-15T20:15:43.375Z
62ac59847bd21e09d0053375
https://cdn-avatars.huggingface.co/v1/production/uploads/1655462260649-noauth.jpeg
Robin Tang
user
LemonPlay
6644fa49123d31ba5b288504
Daniel Zhaohan Guo
6644fa49123d31ba5b288505
Zeyu Zheng
6644fa49123d31ba5b288506
Daniele Calandriello
6644fa49123d31ba5b288507
Yuan Cao
admin_assigned
2024-05-15T20:14:18.573Z
6433a8c37b8247480105cf7f
/avatars/a19536f3f3b8eae3609648bea112b381.svg
Yuan Cao
user
caoyuan33
6644fa49123d31ba5b288508
Eugene Tarassov
claimed_verified
2024-05-31T14:24:09.016Z
63280235bfc72a1d59d029c3
/avatars/85be03ddbe48a0d7a12f8fd95eeb00a1.svg
Eugene Tarassov
user
etarassov
6644fa49123d31ba5b288509
Rémi Munos
6644fa49123d31ba5b28850a
Bernardo Ávila Pires
6644fa49123d31ba5b28850b
Michal Valko
admin_assigned
2024-05-15T20:14:50.193Z
651e97156d92456bdf5ace6b
https://cdn-avatars.huggingface.co/v1/production/uploads/651e97156d92456bdf5ace6b/KKfdZGPAcWPdqycp9SulH.jpeg
Michal Valko
user
misovalko
6644fa49123d31ba5b28850c
Yong Cheng
6644fa49123d31ba5b28850d
Will Dabney
Understanding the performance gap between online and offline alignment
  algorithms
Reinforcement learning from human feedback (RLHF) is the canonical framework
for large language model alignment. However, rising popularity in offline
alignment algorithms challenge the need for on-policy sampling in RLHF. Within
the context of reward over-optimization, we start with an opening set of
experiments that demonstrate the clear advantage of online methods over offline
methods. This prompts us to investigate the causes to the performance
discrepancy through a series of carefully designed experimental ablations. We
show empirically that hypotheses such as offline data coverage and data quality
by itself cannot convincingly explain the performance difference. We also find
that while offline algorithms train policy to become good at pairwise
classification, it is worse at generations; in the meantime the policies
trained by online algorithms are good at generations while worse at pairwise
classification. This hints at a unique interplay between discriminative and
generative capabilities, which is greatly impacted by the sampling process.
Lastly, we observe that the performance discrepancy persists for both
contrastive and non-contrastive loss functions, and appears not to be addressed
by simply scaling up policy networks. Taken together, our study sheds light on
the pivotal role of on-policy sampling in AI alignment, and hints at certain
fundamental challenges of offline alignment algorithms.
Offline reinforcement learning from human feedback underperforms online methods in large language model alignment, suggesting that on-policy sampling is crucial and highlighting challenges in offline alignment.
reinforcement learning from human feedback RLHF offline alignment online methods reward over-optimization experimental ablations offline data coverage data quality pairwise classification generative capabilities contrastive loss non-contrastive loss policy networks AI alignment