Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical
  Reasoning
Direct Preference Optimization (DPO) has proven effective at improving the
performance of large language models (LLMs) on downstream tasks such as
reasoning and alignment. In this work, we propose Step-Controlled DPO (SCDPO),
a method for automatically providing stepwise error supervision by creating
negative samples of mathematical reasoning rationales that start making errors
at a specified step. By applying these samples in DPO training, SCDPO can
better align the model to understand reasoning errors and output accurate
reasoning steps. We apply SCDPO to both code-integrated and chain-of-thought
solutions, empirically showing that it consistently improves the performance
compared to naive DPO on three different SFT models, including one existing SFT
model and two models we finetuned. Qualitative analysis of the credit
assignment of SCDPO and DPO demonstrates the effectiveness of SCDPO at
identifying errors in mathematical solutions. We then apply SCDPO to an
InternLM2-20B model, resulting in a 20B model that achieves high scores of
88.5% on GSM8K and 58.1% on MATH, rivaling all other open-source LLMs, showing
the great potential of our method.
66837739ce294ddc5ecd48c6
Zimu Lu
claimed_verified
2024-07-04T08:04:08.869Z
64b0bfef2f2f9c345b87e673
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/7O3zCr_IyGvtz0BUdKNKi.jpeg
Zimu Lu
user
luzimu
66837739ce294ddc5ecd48c7
Aojun Zhou
claimed_verified
2024-07-03T13:44:54.197Z
637de1520d5bb06fbe5207a9
/avatars/1090851217270c5a858b13e013356d4f.svg
AJ.Zhou
user
AJZhou
66837739ce294ddc5ecd48c8
Ke Wang
claimed_verified
2024-10-11T07:45:38.829Z
64d592c28767727dffa1f002
/avatars/fe38bcac944a2742dc12c624e62d24ef.svg
WangKe
user
scikkk
66837739ce294ddc5ecd48c9
Houxing Ren
admin_assigned
2024-07-03T09:29:51.361Z
63e4b63dd6278c161be47f4b
https://cdn-avatars.huggingface.co/v1/production/uploads/63e4b63dd6278c161be47f4b/u0y1_o9WBqPJh8C8dimJ8.jpeg
Houxing Ren
user
Houxing
66837739ce294ddc5ecd48ca
Weikang Shi
admin_assigned
2024-07-03T09:29:58.440Z
6337e1d1107c4835a04c8607
/avatars/e85353b34c3da6efef84951934811fee.svg
Weikang Shi
user
shiwk20
66837739ce294ddc5ecd48cb
Junting Pan
admin_assigned
2024-07-03T09:30:08.847Z
639c2d36e62c431aa8d834a7
/avatars/a23d57dcb190711494a2a7e75e4cb9a8.svg
Junting Pan
user
juntingpan
66837739ce294ddc5ecd48cc
Mingjie Zhan
Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical
  Reasoning
Direct Preference Optimization (DPO) has proven effective at improving the
performance of large language models (LLMs) on downstream tasks such as
reasoning and alignment. In this work, we propose Step-Controlled DPO (SCDPO),
a method for automatically providing stepwise error supervision by creating
negative samples of mathematical reasoning rationales that start making errors
at a specified step. By applying these samples in DPO training, SCDPO can
better align the model to understand reasoning errors and output accurate
reasoning steps. We apply SCDPO to both code-integrated and chain-of-thought
solutions, empirically showing that it consistently improves the performance
compared to naive DPO on three different SFT models, including one existing SFT
model and two models we finetuned. Qualitative analysis of the credit
assignment of SCDPO and DPO demonstrates the effectiveness of SCDPO at
identifying errors in mathematical solutions. We then apply SCDPO to an
InternLM2-20B model, resulting in a 20B model that achieves high scores of
88.5% on GSM8K and 58.1% on MATH, rivaling all other open-source LLMs, showing
the great potential of our method.
Step-Controlled DPO enhances large language models by providing targeted error supervision, improving reasoning and alignment compared to standard DPO.
Direct Preference Optimization Step-Controlled DPO negative samples mathematical reasoning rationales automatic stepwise error supervision SFT models chain-of-thought solutions credit assignment InternLM2-20B GSM8K MATH