title: Teaching Large Language Models to Reason with Reinforcement Learning
summary: Reinforcement Learning from Human Feedback (RLHF) has emerged as a
  dominant approach for aligning LLM outputs with human preferences. Inspired by
  the success of RLHF, we study the performance of multiple algorithms that learn
  from feedback (Expert Iteration, Proximal Policy Optimization (PPO),
  Return-Conditioned RL) on improving LLM reasoning capabilities. We investigate
  both sparse and dense rewards provided to the LLM both heuristically and via a
  learned reward model. We additionally start from multiple model sizes and
  initializations both with and without supervised fine-tuning (SFT)
  data. Overall, we find all algorithms perform comparably, with Expert Iteration
  performing best in most cases. Surprisingly, we find the sample complexity of
  Expert Iteration is similar to that of PPO, requiring at most on the order of
  10^6 samples to converge from a pretrained checkpoint. We investigate why
  this is the case, concluding that during RL training models fail to explore
  significantly beyond solutions already produced by SFT models. Additionally, we
  discuss a trade off between maj@1 and pass@96 metric performance during SFT
  training and how conversely RL training improves both simultaneously. We then
  conclude by discussing the implications of our findings for RLHF and the future
  role of RL in LLM fine-tuning.
paper_authors._id: 65ea7cba8d4052d338ab202c
paper_authors.name: Alex Havrilla
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:13:42.441Z
paper_authors._id: 6144d541fa30a4ba3ef7ef15
paper_authors.avatarUrl: /avatars/eeb849eb29e249fd1b40e5c10d9bf9e5.svg
paper_authors.fullname: Alex Havrilla
paper_authors.type: user
paper_authors.user: Dahoas
paper_authors._id: 65ea7cba8d4052d338ab202d
paper_authors.name: Yuqing Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:13:49.773Z
paper_authors._id: 64b7915aa8c39dc07887e035
paper_authors.avatarUrl: /avatars/271bd18eb7d4b5bae8d71674ccb3a792.svg
paper_authors.fullname: Yuqing Du
paper_authors.type: user
paper_authors.user: yuqingd
paper_authors._id: 65ea7cba8d4052d338ab202e
paper_authors.name: Sharath Chandra Raparthy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:14:01.734Z
paper_authors._id: 65007a96ea5699b59a9bb432
paper_authors.avatarUrl: /avatars/4549f5964b007de189c45a77daf10075.svg
paper_authors.fullname: Sharath Raparthy
paper_authors.type: user
paper_authors.user: sharathraparthy
paper_authors._id: 65ea7cba8d4052d338ab202f
paper_authors.name: Christoforos Nalmpantis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:14:08.341Z
paper_authors._id: 627e1e4f9632ca4815f0a729
paper_authors.avatarUrl: /avatars/2ba615cdc1d9e3eb0397366cff7a7c8b.svg
paper_authors.fullname: Christoforos Nalmpantis
paper_authors.type: user
paper_authors.user: christofernal
paper_authors._id: 65ea7cba8d4052d338ab2030
paper_authors.name: Jane Dwivedi-Yu
paper_authors._id: 65ea7cba8d4052d338ab2031
paper_authors.name: Maksym Zhuravinskyi
paper_authors._id: 65ea7cba8d4052d338ab2032
paper_authors.name: Eric Hambro
paper_authors._id: 65ea7cba8d4052d338ab2033
paper_authors.name: Sainbayar Sukhbaatar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-30T16:01:24.362Z
paper_authors._id: 66a8611eb51510d82ed54231
paper_authors.avatarUrl: /avatars/ad559e774fee4914091b82c9831ae2a2.svg
paper_authors.fullname: Sainbayar Sukhbaatar
paper_authors.type: user
paper_authors.user: sainbar
paper_authors._id: 65ea7cba8d4052d338ab2034
paper_authors.name: Roberta Raileanu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:14:42.769Z
paper_authors._id: 633e94793a17ab61de8e2b9c
paper_authors.avatarUrl: /avatars/5f2f58ddeed211393660ada6b135f0d5.svg
paper_authors.fullname: Roberta Raileanu
paper_authors.type: user
paper_authors.user: rraileanu
paper_title: Teaching Large Language Models to Reason with Reinforcement Learning
paper_summary: Reinforcement Learning from Human Feedback (RLHF) has emerged as a
  dominant approach for aligning LLM outputs with human preferences. Inspired by
  the success of RLHF, we study the performance of multiple algorithms that learn
  from feedback (Expert Iteration, Proximal Policy Optimization (PPO),
  Return-Conditioned RL) on improving LLM reasoning capabilities. We investigate
  both sparse and dense rewards provided to the LLM both heuristically and via a
  learned reward model. We additionally start from multiple model sizes and
  initializations both with and without supervised fine-tuning (SFT)
  data. Overall, we find all algorithms perform comparably, with Expert Iteration
  performing best in most cases. Surprisingly, we find the sample complexity of
  Expert Iteration is similar to that of PPO, requiring at most on the order of
  10^6 samples to converge from a pretrained checkpoint. We investigate why
  this is the case, concluding that during RL training models fail to explore
  significantly beyond solutions already produced by SFT models. Additionally, we
  discuss a trade off between maj@1 and pass@96 metric performance during SFT
  training and how conversely RL training improves both simultaneously. We then
  conclude by discussing the implications of our findings for RLHF and the future
  role of RL in LLM fine-tuning.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) Expert Iteration Proximal Policy Optimization (PPO) Return-Conditioned RL sparse rewards dense rewards learned reward model supervised fine-tuning (SFT)