title: Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of
    LLMs
summary: Mathematical reasoning presents a significant challenge for Large Language
  Models (LLMs) due to the extensive and precise chain of reasoning required for
  accuracy. Ensuring the correctness of each reasoning step is critical. To
  address this, we aim to enhance the robustness and factuality of LLMs by
  learning from human feedback. However, Direct Preference Optimization (DPO) has
  shown limited benefits for long-chain mathematical reasoning, as models
  employing DPO struggle to identify detailed errors in incorrect answers. This
  limitation stems from a lack of fine-grained process supervision. We propose a
  simple, effective, and data-efficient method called Step-DPO, which treats
  individual reasoning steps as units for preference optimization rather than
  evaluating answers holistically. Additionally, we have developed a data
  construction pipeline for Step-DPO, enabling the creation of a high-quality
  dataset containing 10K step-wise preference pairs. We also observe that in DPO,
  self-generated data is more effective than data generated by humans or GPT-4,
  due to the latter's out-of-distribution nature. Our findings demonstrate that
  as few as 10K preference data pairs and fewer than 500 Step-DPO training steps
  can yield a nearly 3% gain in accuracy on MATH for models with over 70B
  parameters. Notably, Step-DPO, when applied to Qwen2-72B-Instruct, achieves
  scores of 70.8% and 94.0% on the test sets of MATH and GSM8K, respectively,
  surpassing a series of closed-source models, including GPT-4-1106,
  Claude-3-Opus, and Gemini-1.5-Pro. Our code, data, and models are available at
  https://github.com/dvlab-research/Step-DPO.
paper_authors._id: 667e1ec7b3072cb50c79ef59
paper_authors.name: Xin Lai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:53:21.974Z
paper_authors._id: 642bddc1fc41757877f68327
paper_authors.avatarUrl: /avatars/f275237f36a112624d59a7e3f73237d3.svg
paper_authors.fullname: Xin Lai
paper_authors.type: user
paper_authors.user: xinlai
paper_authors._id: 667e1ec7b3072cb50c79ef5a
paper_authors.name: Zhuotao Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:53:16.366Z
paper_authors._id: 64d104a37a7305c5895bd720
paper_authors.avatarUrl: /avatars/2d9eff3a2dff6d02e45ae8964fb91f27.svg
paper_authors.fullname: zt tian
paper_authors.type: user
paper_authors.user: tianzhuotao
paper_authors._id: 667e1ec7b3072cb50c79ef5b
paper_authors.name: Yukang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:53:04.262Z
paper_authors._id: 65b31d77f32c79fea79ba4d1
paper_authors.avatarUrl: /avatars/8c51c1ff337ac372c7caafa4fa5019dc.svg
paper_authors.fullname: Yukang Chen
paper_authors.type: user
paper_authors.user: Yukang2
paper_authors._id: 667e1ec7b3072cb50c79ef5c
paper_authors.name: Senqiao Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:52:57.935Z
paper_authors._id: 6527b7280ae663e384eb8499
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6527b7280ae663e384eb8499/73yF3eu2cUx7jVZrhXnXx.jpeg
paper_authors.fullname: Senqiao Yang
paper_authors.type: user
paper_authors.user: Senqiao
paper_authors._id: 667e1ec7b3072cb50c79ef5d
paper_authors.name: Xiangru Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:52:52.509Z
paper_authors._id: 667e3aea97e3c9e6086a861f
paper_authors.avatarUrl: /avatars/22d66b872173ce0935c2f4e6ba1e001d.svg
paper_authors.fullname: Xiangru Peng
paper_authors.type: user
paper_authors.user: pengxr
paper_authors._id: 667e1ec7b3072cb50c79ef5e
paper_authors.name: Jiaya Jia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:52:46.492Z
paper_authors._id: 651ff0416f1df5466366400d
paper_authors.avatarUrl: /avatars/caba60f52a490fc3aece6d4ae036d2c3.svg
paper_authors.fullname: Jim
paper_authors.type: user
paper_authors.user: jiajiaya
paper_title: Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of
    LLMs
paper_summary: Mathematical reasoning presents a significant challenge for Large Language
  Models (LLMs) due to the extensive and precise chain of reasoning required for
  accuracy. Ensuring the correctness of each reasoning step is critical. To
  address this, we aim to enhance the robustness and factuality of LLMs by
  learning from human feedback. However, Direct Preference Optimization (DPO) has
  shown limited benefits for long-chain mathematical reasoning, as models
  employing DPO struggle to identify detailed errors in incorrect answers. This
  limitation stems from a lack of fine-grained process supervision. We propose a
  simple, effective, and data-efficient method called Step-DPO, which treats
  individual reasoning steps as units for preference optimization rather than
  evaluating answers holistically. Additionally, we have developed a data
  construction pipeline for Step-DPO, enabling the creation of a high-quality
  dataset containing 10K step-wise preference pairs. We also observe that in DPO,
  self-generated data is more effective than data generated by humans or GPT-4,
  due to the latter's out-of-distribution nature. Our findings demonstrate that
  as few as 10K preference data pairs and fewer than 500 Step-DPO training steps
  can yield a nearly 3% gain in accuracy on MATH for models with over 70B
  parameters. Notably, Step-DPO, when applied to Qwen2-72B-Instruct, achieves
  scores of 70.8% and 94.0% on the test sets of MATH and GSM8K, respectively,
  surpassing a series of closed-source models, including GPT-4-1106,
  Claude-3-Opus, and Gemini-1.5-Pro. Our code, data, and models are available at
  https://github.com/dvlab-research/Step-DPO.
paper_ai_keywords: Large Language Models (LLMs) Direct Preference Optimization (DPO) Step-DPO preference optimization reasoning steps high-quality dataset preference pairs MATH GSM8K Qwen2-72B-Instruct GPT-4-1106 Claude-3-Opus Gemini-1.5-Pro