title: SLiC-HF: Sequence Likelihood Calibration with Human Feedback
summary: Learning from human feedback has been shown to be effective at aligning
  language models with human preferences. Past work has often relied on
  Reinforcement Learning from Human Feedback (RLHF), which optimizes the language
  model using reward scores assigned from a reward model trained on human
  preference data. In this work we show how the recently introduced Sequence
  Likelihood Calibration (SLiC), can also be used to effectively learn from human
  preferences (SLiC-HF). Furthermore, we demonstrate this can be done with human
  feedback data collected for a different model, similar to off-policy, offline
  RL data. Automatic and human evaluation experiments on the TL;DR summarization
  task show that SLiC-HF significantly improves supervised fine-tuning baselines.
  Furthermore, SLiC-HF presents a competitive alternative to the PPO RLHF
  implementation used in past work while being much simpler to implement, easier
  to tune and more computationally efficient in practice.
paper_authors._id: 6465885e34acce85aea4dde7
paper_authors.name: Yao Zhao
paper_authors._id: 6465885e34acce85aea4dde8
paper_authors.name: Rishabh Joshi
paper_authors._id: 6465885e34acce85aea4dde9
paper_authors.name: Tianqi Liu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-05-19T14:17:57.179Z
paper_authors._id: 646784af696e7355f5d5001f
paper_authors.avatarUrl: /avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
paper_authors.fullname: Tianqi Liu
paper_authors.type: user
paper_authors.user: TianqiLiuAI
paper_authors._id: 6465885e34acce85aea4ddea
paper_authors.name: Misha Khalman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-07T20:41:29.156Z
paper_authors._id: 65c2ad2790703d2dbcf79471
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65c2ad2790703d2dbcf79471/H15_ASSI7fCFDCDF4xgIb.jpeg
paper_authors.fullname: Misha Khalman
paper_authors.type: user
paper_authors.user: khalman
paper_authors._id: 6465885e34acce85aea4ddeb
paper_authors.name: Mohammad Saleh
paper_authors._id: 6465885e34acce85aea4ddec
paper_authors.name: Peter J. Liu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-05-18T02:08:56.065Z
paper_authors._id: 62ded294dcdc9043efaa4fc3
paper_authors.avatarUrl: /avatars/1294078b8cd5c1901e4021253d877357.svg
paper_authors.fullname: Peter J Liu
paper_authors.type: user
paper_authors.user: peterjliu
paper_title: SLiC-HF: Sequence Likelihood Calibration with Human Feedback
paper_summary: Learning from human feedback has been shown to be effective at aligning
  language models with human preferences. Past work has often relied on
  Reinforcement Learning from Human Feedback (RLHF), which optimizes the language
  model using reward scores assigned from a reward model trained on human
  preference data. In this work we show how the recently introduced Sequence
  Likelihood Calibration (SLiC), can also be used to effectively learn from human
  preferences (SLiC-HF). Furthermore, we demonstrate this can be done with human
  feedback data collected for a different model, similar to off-policy, offline
  RL data. Automatic and human evaluation experiments on the TL;DR summarization
  task show that SLiC-HF significantly improves supervised fine-tuning baselines.
  Furthermore, SLiC-HF presents a competitive alternative to the PPO RLHF
  implementation used in past work while being much simpler to implement, easier
  to tune and more computationally efficient in practice.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) Sequence Likelihood Calibration (SLiC) Reward model Human preference data Multimodal reinforcement learning Off-policy reinforcement learning Offline reinforcement learning data TL;DR summarization Supervised fine-tuning PPO reinforcement learning