title: Direct Language Model Alignment from Online AI Feedback
summary: Direct alignment from preferences (DAP) methods, such as DPO, have recently
  emerged as efficient alternatives to reinforcement learning from human feedback
  (RLHF), that do not require a separate reward model. However, the preference
  datasets used in DAP methods are usually collected ahead of training and never
  updated, thus the feedback is purely offline. Moreover, responses in these
  datasets are often sampled from a language model distinct from the one being
  aligned, and since the model evolves over training, the alignment phase is
  inevitably off-policy. In this study, we posit that online feedback is key and
  improves DAP methods. Our method, online AI feedback (OAIF), uses an LLM as
  annotator: on each training iteration, we sample two responses from the current
  model and prompt the LLM annotator to choose which one is preferred, thus
  providing online feedback. Despite its simplicity, we demonstrate via human
  evaluation in several tasks that OAIF outperforms both offline DAP and RLHF
  methods. We further show that the feedback leveraged in OAIF is easily
  controllable, via instruction prompts to the LLM annotator.
paper_authors._id: 65c43fca7273e436da4ace13
paper_authors.name: Shangmin Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-08T11:52:50.223Z
paper_authors._id: 64406aac7841867cd5b5b9fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64406aac7841867cd5b5b9fa/hPPiqtbYv-42q8LOwB22p.png
paper_authors.fullname: Shawn Guo
paper_authors.type: user
paper_authors.user: Shawn-Guo
paper_authors._id: 65c43fca7273e436da4ace14
paper_authors.name: Biao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-09T08:05:51.869Z
paper_authors._id: 6346a600bcb5e67902ac6f44
paper_authors.avatarUrl: /avatars/cf42043c19990bb388223a61c7a1e292.svg
paper_authors.fullname: Biao Zhang
paper_authors.type: user
paper_authors.user: bzhanggo
paper_authors._id: 65c43fca7273e436da4ace15
paper_authors.name: Tianlin Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:10:49.055Z
paper_authors._id: 64268d8d2f16fd599d9e9576
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64268d8d2f16fd599d9e9576/yTWuz4wQbXD6QlovBY6-M.jpeg
paper_authors.fullname: Tianlin Liu
paper_authors.type: user
paper_authors.user: tianlinliu0121
paper_authors._id: 65c43fca7273e436da4ace16
paper_authors.name: Tianqi Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-29T08:40:54.017Z
paper_authors._id: 646784af696e7355f5d5001f
paper_authors.avatarUrl: /avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
paper_authors.fullname: Tianqi Liu
paper_authors.type: user
paper_authors.user: TianqiLiuAI
paper_authors._id: 65c43fca7273e436da4ace17
paper_authors.name: Misha Khalman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:11:00.617Z
paper_authors._id: 65c2ad2790703d2dbcf79471
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65c2ad2790703d2dbcf79471/H15_ASSI7fCFDCDF4xgIb.jpeg
paper_authors.fullname: Misha Khalman
paper_authors.type: user
paper_authors.user: khalman
paper_authors._id: 65c43fca7273e436da4ace18
paper_authors.name: Felipe Llinares
paper_authors._id: 65c43fca7273e436da4ace19
paper_authors.name: Alexandre Rame
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:11:13.950Z
paper_authors._id: 63c94ede00104ea998de19a6
paper_authors.avatarUrl: /avatars/273959d87f0c67747588cf0700d64039.svg
paper_authors.fullname: Alexandre Rame
paper_authors.type: user
paper_authors.user: alexrame
paper_authors._id: 65c43fca7273e436da4ace1a
paper_authors.name: Thomas Mesnard
paper_authors._id: 65c43fca7273e436da4ace1b
paper_authors.name: Yao Zhao
paper_authors._id: 65c43fca7273e436da4ace1c
paper_authors.name: Bilal Piot
paper_authors._id: 65c43fca7273e436da4ace1d
paper_authors.name: Johan Ferret
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:11:45.967Z
paper_authors._id: 65afb7dbdd6bdfd73cd8e609
paper_authors.avatarUrl: /avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg
paper_authors.fullname: Johan Ferret
paper_authors.type: user
paper_authors.user: ferretj
paper_authors._id: 65c43fca7273e436da4ace1e
paper_authors.name: Mathieu Blondel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T16:46:53.081Z
paper_authors._id: 66d093b681e0683bca48bed6
paper_authors.avatarUrl: /avatars/cc1fbccb0b6aa93d648bcbdf9c3a35e1.svg
paper_authors.fullname: Mathieu Blondel
paper_authors.type: user
paper_authors.user: mblondel
paper_title: Direct Language Model Alignment from Online AI Feedback
paper_summary: Direct alignment from preferences (DAP) methods, such as DPO, have recently
  emerged as efficient alternatives to reinforcement learning from human feedback
  (RLHF), that do not require a separate reward model. However, the preference
  datasets used in DAP methods are usually collected ahead of training and never
  updated, thus the feedback is purely offline. Moreover, responses in these
  datasets are often sampled from a language model distinct from the one being
  aligned, and since the model evolves over training, the alignment phase is
  inevitably off-policy. In this study, we posit that online feedback is key and
  improves DAP methods. Our method, online AI feedback (OAIF), uses an LLM as
  annotator: on each training iteration, we sample two responses from the current
  model and prompt the LLM annotator to choose which one is preferred, thus
  providing online feedback. Despite its simplicity, we demonstrate via human
  evaluation in several tasks that OAIF outperforms both offline DAP and RLHF
  methods. We further show that the feedback leveraged in OAIF is easily
  controllable, via instruction prompts to the LLM annotator.
paper_ai_summary: Online AI feedback improves direct alignment from preferences methods by providing real-time feedback from a model, outperforming both offline DAP and RLHF methods.
paper_ai_keywords: direct alignment from preferences DPO reinforcement learning from human feedback preference datasets offline feedback on-policy off-policy online AI feedback LLM annotator human evaluation