title: Iterative Reasoning Preference Optimization
summary: Iterative preference optimization methods have recently been shown to perform
  well for general instruction tuning tasks, but typically make little
  improvement on reasoning tasks (Yuan et al., 2024, Chen et al., 2024). In this
  work we develop an iterative approach that optimizes the preference between
  competing generated Chain-of-Thought (CoT) candidates by optimizing for winning
  vs. losing reasoning steps that lead to the correct answer. We train using a
  modified DPO loss (Rafailov et al., 2023) with an additional negative
  log-likelihood term, which we find to be crucial. We show reasoning improves
  across repeated iterations of this scheme. While only relying on examples in
  the training set, our approach results in increasing accuracy for
  Llama-2-70B-Chat from 55.6% to 81.6% on GSM8K (and 88.7% with majority voting
  out of 32 samples), from 12.5% to 20.8% on MATH, and from 77.8% to 86.7% on
  ARC-Challenge, which outperforms other Llama-2-based models not relying on
  additionally sourced datasets.
paper_authors._id: 6631a5d3992449cefe049ad9
paper_authors.name: Richard Yuanzhe Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T08:11:34.986Z
paper_authors._id: 64d50fa59a6a7ae984d90852
paper_authors.avatarUrl: /avatars/26b78c151e707e3f9a9516c877d18ff7.svg
paper_authors.fullname: Richard Pang
paper_authors.type: user
paper_authors.user: yzpang
paper_authors._id: 6631a5d3992449cefe049ada
paper_authors.name: Weizhe Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T08:11:41.658Z
paper_authors._id: 6172aa78c8e66e2aa84c06b8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1634904686602-noauth.png
paper_authors.fullname: Weizhe Yuan
paper_authors.type: user
paper_authors.user: weizhey
paper_authors._id: 6631a5d3992449cefe049adb
paper_authors.name: Kyunghyun Cho
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T08:11:49.259Z
paper_authors._id: 649c1b8485b05ee02723334a
paper_authors.avatarUrl: /avatars/29080f550fdb471c4d2daace86aff3b0.svg
paper_authors.fullname: Kyunghyun Cho
paper_authors.type: user
paper_authors.user: kyunghyuncho
paper_authors._id: 6631a5d3992449cefe049adc
paper_authors.name: He He
paper_authors._id: 6631a5d3992449cefe049add
paper_authors.name: Sainbayar Sukhbaatar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-30T16:01:18.113Z
paper_authors._id: 66a8611eb51510d82ed54231
paper_authors.avatarUrl: /avatars/ad559e774fee4914091b82c9831ae2a2.svg
paper_authors.fullname: Sainbayar Sukhbaatar
paper_authors.type: user
paper_authors.user: sainbar
paper_authors._id: 6631a5d3992449cefe049ade
paper_authors.name: Jason Weston
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T08:12:02.437Z
paper_authors._id: 62f023a36a027498eaa2f9cc
paper_authors.avatarUrl: /avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
paper_authors.fullname: Jason Weston
paper_authors.type: user
paper_authors.user: spermwhale
paper_title: Iterative Reasoning Preference Optimization
paper_summary: Iterative preference optimization methods have recently been shown to perform
  well for general instruction tuning tasks, but typically make little
  improvement on reasoning tasks (Yuan et al., 2024, Chen et al., 2024). In this
  work we develop an iterative approach that optimizes the preference between
  competing generated Chain-of-Thought (CoT) candidates by optimizing for winning
  vs. losing reasoning steps that lead to the correct answer. We train using a
  modified DPO loss (Rafailov et al., 2023) with an additional negative
  log-likelihood term, which we find to be crucial. We show reasoning improves
  across repeated iterations of this scheme. While only relying on examples in
  the training set, our approach results in increasing accuracy for
  Llama-2-70B-Chat from 55.6% to 81.6% on GSM8K (and 88.7% with majority voting
  out of 32 samples), from 12.5% to 20.8% on MATH, and from 77.8% to 86.7% on
  ARC-Challenge, which outperforms other Llama-2-based models not relying on
  additionally sourced datasets.
paper_ai_summary: An iterative preference optimization method using a modified DPO loss improves reasoning accuracy on various datasets by optimizing winning and losing reasoning steps in Chain-of-Thought candidates.
paper_ai_keywords: iterative preference optimization Chain-of-Thought (CoT) DPO loss negative log-likelihood term GSM8K MATH ARC-Challenge