title: Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical
    Reasoning
summary: Direct Preference Optimization (DPO) has proven effective at improving the
  performance of large language models (LLMs) on downstream tasks such as
  reasoning and alignment. In this work, we propose Step-Controlled DPO (SCDPO),
  a method for automatically providing stepwise error supervision by creating
  negative samples of mathematical reasoning rationales that start making errors
  at a specified step. By applying these samples in DPO training, SCDPO can
  better align the model to understand reasoning errors and output accurate
  reasoning steps. We apply SCDPO to both code-integrated and chain-of-thought
  solutions, empirically showing that it consistently improves the performance
  compared to naive DPO on three different SFT models, including one existing SFT
  model and two models we finetuned. Qualitative analysis of the credit
  assignment of SCDPO and DPO demonstrates the effectiveness of SCDPO at
  identifying errors in mathematical solutions. We then apply SCDPO to an
  InternLM2-20B model, resulting in a 20B model that achieves high scores of
  88.5% on GSM8K and 58.1% on MATH, rivaling all other open-source LLMs, showing
  the great potential of our method.
paper_authors._id: 66837739ce294ddc5ecd48c6
paper_authors.name: Zimu Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-04T08:04:08.869Z
paper_authors._id: 64b0bfef2f2f9c345b87e673
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/7O3zCr_IyGvtz0BUdKNKi.jpeg
paper_authors.fullname: Zimu Lu
paper_authors.type: user
paper_authors.user: luzimu
paper_authors._id: 66837739ce294ddc5ecd48c7
paper_authors.name: Aojun Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-03T13:44:54.197Z
paper_authors._id: 637de1520d5bb06fbe5207a9
paper_authors.avatarUrl: /avatars/1090851217270c5a858b13e013356d4f.svg
paper_authors.fullname: AJ.Zhou
paper_authors.type: user
paper_authors.user: AJZhou
paper_authors._id: 66837739ce294ddc5ecd48c8
paper_authors.name: Ke Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T07:45:38.829Z
paper_authors._id: 64d592c28767727dffa1f002
paper_authors.avatarUrl: /avatars/fe38bcac944a2742dc12c624e62d24ef.svg
paper_authors.fullname: WangKe
paper_authors.type: user
paper_authors.user: scikkk
paper_authors._id: 66837739ce294ddc5ecd48c9
paper_authors.name: Houxing Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T09:29:51.361Z
paper_authors._id: 63e4b63dd6278c161be47f4b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e4b63dd6278c161be47f4b/u0y1_o9WBqPJh8C8dimJ8.jpeg
paper_authors.fullname: Houxing Ren
paper_authors.type: user
paper_authors.user: Houxing
paper_authors._id: 66837739ce294ddc5ecd48ca
paper_authors.name: Weikang Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T09:29:58.440Z
paper_authors._id: 6337e1d1107c4835a04c8607
paper_authors.avatarUrl: /avatars/e85353b34c3da6efef84951934811fee.svg
paper_authors.fullname: Weikang Shi
paper_authors.type: user
paper_authors.user: shiwk20
paper_authors._id: 66837739ce294ddc5ecd48cb
paper_authors.name: Junting Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T09:30:08.847Z
paper_authors._id: 639c2d36e62c431aa8d834a7
paper_authors.avatarUrl: /avatars/a23d57dcb190711494a2a7e75e4cb9a8.svg
paper_authors.fullname: Junting Pan
paper_authors.type: user
paper_authors.user: juntingpan
paper_authors._id: 66837739ce294ddc5ecd48cc
paper_authors.name: Mingjie Zhan
paper_title: Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical
    Reasoning
paper_summary: Direct Preference Optimization (DPO) has proven effective at improving the
  performance of large language models (LLMs) on downstream tasks such as
  reasoning and alignment. In this work, we propose Step-Controlled DPO (SCDPO),
  a method for automatically providing stepwise error supervision by creating
  negative samples of mathematical reasoning rationales that start making errors
  at a specified step. By applying these samples in DPO training, SCDPO can
  better align the model to understand reasoning errors and output accurate
  reasoning steps. We apply SCDPO to both code-integrated and chain-of-thought
  solutions, empirically showing that it consistently improves the performance
  compared to naive DPO on three different SFT models, including one existing SFT
  model and two models we finetuned. Qualitative analysis of the credit
  assignment of SCDPO and DPO demonstrates the effectiveness of SCDPO at
  identifying errors in mathematical solutions. We then apply SCDPO to an
  InternLM2-20B model, resulting in a 20B model that achieves high scores of
  88.5% on GSM8K and 58.1% on MATH, rivaling all other open-source LLMs, showing
  the great potential of our method.
paper_ai_keywords: Direct Preference Optimization (DPO) Step-Controlled DPO (SCDPO) stepwise error supervision mathematical reasoning rationales multi-negative samples alignment reasoning errors reasoning steps code-integrated chain-of-thought solutions SFT models qualitative analysis credit assignment InternLM2-20B GSM8K MATH