title: Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary
    Feedback
summary: Large language models (LLMs) have recently demonstrated remarkable success in
  mathematical reasoning. Despite progress in methods like chain-of-thought
  prompting and self-consistency sampling, these advances often focus on final
  correctness without ensuring that the underlying reasoning process is coherent
  and reliable. This paper introduces Step-KTO, a training framework that
  combines process-level and outcome-level binary feedback to guide LLMs toward
  more trustworthy reasoning trajectories. By providing binary evaluations for
  both the intermediate reasoning steps and the final answer, Step-KTO encourages
  the model to adhere to logical progressions rather than relying on superficial
  shortcuts. Our experiments on challenging mathematical benchmarks show that
  Step-KTO significantly improves both final answer accuracy and the quality of
  intermediate reasoning steps. For example, on the MATH-500 dataset, Step-KTO
  achieves a notable improvement in Pass@1 accuracy over strong baselines. These
  results highlight the promise of integrating stepwise process feedback into LLM
  training, paving the way toward more interpretable and dependable reasoning
  capabilities.
paper_authors._id: 679208664e521de952ca0cdc
paper_authors.name: Yen-Ting Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-23T09:17:30.742Z
paper_authors._id: 5df9c78eda6d0311fd3d541f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5df9c78eda6d0311fd3d541f/8oDFuP77l5zhvamXNVmnc.jpeg
paper_authors.fullname: Yen-Ting Lin
paper_authors.type: user
paper_authors.user: yentinglin
paper_authors._id: 679208664e521de952ca0cdd
paper_authors.name: Di Jin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:15:36.716Z
paper_authors._id: 62f690dfc58915315c504af5
paper_authors.avatarUrl: /avatars/a6732dda8cd7e37d9c0e0b1dfb68c66b.svg
paper_authors.fullname: Di Jin
paper_authors.type: user
paper_authors.user: jindi
paper_authors._id: 679208664e521de952ca0cde
paper_authors.name: Tengyu Xu
paper_authors._id: 679208664e521de952ca0cdf
paper_authors.name: Tianhao Wu
paper_authors._id: 679208664e521de952ca0ce0
paper_authors.name: Sainbayar Sukhbaatar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:16:23.763Z
paper_authors._id: 66a8611eb51510d82ed54231
paper_authors.avatarUrl: /avatars/ad559e774fee4914091b82c9831ae2a2.svg
paper_authors.fullname: Sainbayar Sukhbaatar
paper_authors.type: user
paper_authors.user: sainbar
paper_authors._id: 679208664e521de952ca0ce1
paper_authors.name: Chen Zhu
paper_authors._id: 679208664e521de952ca0ce2
paper_authors.name: Yun He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:16:34.273Z
paper_authors._id: 6437de5d51c7ebfc813ce68a
paper_authors.avatarUrl: /avatars/144cb1c5d5a4a645080611953494f437.svg
paper_authors.fullname: he
paper_authors.type: user
paper_authors.user: yunhe
paper_authors._id: 679208664e521de952ca0ce3
paper_authors.name: Yun-Nung Chen
paper_authors._id: 679208664e521de952ca0ce4
paper_authors.name: Jason Weston
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:15:25.564Z
paper_authors._id: 62f023a36a027498eaa2f9cc
paper_authors.avatarUrl: /avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
paper_authors.fullname: Jason Weston
paper_authors.type: user
paper_authors.user: spermwhale
paper_authors._id: 679208664e521de952ca0ce5
paper_authors.name: Yuandong Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:16:47.602Z
paper_authors._id: 6344cf73ee1504dbcd5bdfe7
paper_authors.avatarUrl: /avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
paper_authors.fullname: Yuandong Tian
paper_authors.type: user
paper_authors.user: tydsh
paper_authors._id: 679208664e521de952ca0ce6
paper_authors.name: Arash Rahnama
paper_authors._id: 679208664e521de952ca0ce7
paper_authors.name: Sinong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:17:03.211Z
paper_authors._id: 65b483c5ed110eb9f1ee62df
paper_authors.avatarUrl: /avatars/29100098f5aed1735675d06c516a85b7.svg
paper_authors.fullname: Theron S. Wang
paper_authors.type: user
paper_authors.user: TheronWong
paper_authors._id: 679208664e521de952ca0ce8
paper_authors.name: Hao Ma
paper_authors._id: 679208664e521de952ca0ce9
paper_authors.name: Han Fang
paper_title: Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary
    Feedback
paper_summary: Large language models (LLMs) have recently demonstrated remarkable success in
  mathematical reasoning. Despite progress in methods like chain-of-thought
  prompting and self-consistency sampling, these advances often focus on final
  correctness without ensuring that the underlying reasoning process is coherent
  and reliable. This paper introduces Step-KTO, a training framework that
  combines process-level and outcome-level binary feedback to guide LLMs toward
  more trustworthy reasoning trajectories. By providing binary evaluations for
  both the intermediate reasoning steps and the final answer, Step-KTO encourages
  the model to adhere to logical progressions rather than relying on superficial
  shortcuts. Our experiments on challenging mathematical benchmarks show that
  Step-KTO significantly improves both final answer accuracy and the quality of
  intermediate reasoning steps. For example, on the MATH-500 dataset, Step-KTO
  achieves a notable improvement in Pass@1 accuracy over strong baselines. These
  results highlight the promise of integrating stepwise process feedback into LLM
  training, paving the way toward more interpretable and dependable reasoning
  capabilities.
paper_ai_summary: Step-KTO, a training framework providing binary feedback on both intermediate steps and final answers, enhances the reasoning coherence and accuracy of large language models.
paper_ai_keywords: chain-of-thought prompting self-consistency sampling Step-KTO process-level feedback outcome-level feedback trustworthiness reasoning trajectories logical progressions MATH-500 Pass@1 accuracy interpretable reasoning dependable reasoning