title: BoostStep: Boosting mathematical capability of Large Language Models via
    improved single-step reasoning
summary: Cutting-edge large language models (LLMs) demonstrate promising performance
  in solving complex math problems with a divide-and-conquer pipeline and the
  assistance of in-context learning (ICL) examples. However, their potential for
  improvement is limited by two critical problems within their ICL examples:
  granularity-mismatch and the ensuing negative-effect noise problem.
  Specifically, the LLMs are capable of the dividing process yet mostly failed by
  inaccurate reasoning within a few conquer steps, while the ICL examples
  retrieved in question-grained sometimes lack relevant steps for a specific
  challenging reasoning step. Further, this disconnect may hinder the correct
  reasoning due to its irrelevance. To this end, we focus on improving the
  reasoning quality within each step and present BoostStep. BoostStep aligns the
  granularity between the retrieving and reasoning on step grained, and provides
  highly related ICL examples for each reasoning step with a novel `first-try'
  strategy. BoostStep provides more relevant examples than the coarse
  question-grained strategy, enhancing the model reasoning quality within each
  step steadily. BoostStep is a general and robust reasoning-enhancing method
  that not only improves standalone reasoning performance but also integrates
  seamlessly with Monte Carlo Tree Search methods (MCTS) to refine both candidate
  generation and decision-making. Quantitatively, it improves GPT-4o and
  Qwen2.5-Math-72B by 3.6\% and 2.0\% respectively on various mathematical
  benchmarks, and 7.5\% gain combined with MCTS.
paper_authors._id: 677cdcd50604b68871999e0f
paper_authors.name: Beichen Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T09:37:59.472Z
paper_authors._id: 64b93578ee257c3a4cfceed1
paper_authors.avatarUrl: /avatars/e6188562254f75a09b4048b800860016.svg
paper_authors.fullname: Beichen Zhang
paper_authors.type: user
paper_authors.user: BeichenZhang
paper_authors._id: 677cdcd50604b68871999e10
paper_authors.name: Yuhong Liu
paper_authors._id: 677cdcd50604b68871999e11
paper_authors.name: Xiaoyi Dong
paper_authors._id: 677cdcd50604b68871999e12
paper_authors.name: Yuhang Zang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T08:41:16.697Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 677cdcd50604b68871999e13
paper_authors.name: Pan Zhang
paper_authors._id: 677cdcd50604b68871999e14
paper_authors.name: Haodong Duan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T08:57:58.362Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 677cdcd50604b68871999e15
paper_authors.name: Yuhang Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T08:57:51.041Z
paper_authors._id: 65000bef18830fabea469fdd
paper_authors.avatarUrl: /avatars/b320c77dfad039d9f9c54127f610d44f.svg
paper_authors.fullname: Cao Yuhang
paper_authors.type: user
paper_authors.user: yhcao
paper_authors._id: 677cdcd50604b68871999e16
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T08:56:41.557Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 677cdcd50604b68871999e17
paper_authors.name: Jiaqi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T09:37:28.234Z
paper_authors._id: 64b4eec4faa3181a5eab9c46
paper_authors.avatarUrl: /avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
paper_authors.fullname: Jiaqi Wang
paper_authors.type: user
paper_authors.user: myownskyW7
paper_title: BoostStep: Boosting mathematical capability of Large Language Models via
    improved single-step reasoning
paper_summary: Cutting-edge large language models (LLMs) demonstrate promising performance
  in solving complex math problems with a divide-and-conquer pipeline and the
  assistance of in-context learning (ICL) examples. However, their potential for
  improvement is limited by two critical problems within their ICL examples:
  granularity-mismatch and the ensuing negative-effect noise problem.
  Specifically, the LLMs are capable of the dividing process yet mostly failed by
  inaccurate reasoning within a few conquer steps, while the ICL examples
  retrieved in question-grained sometimes lack relevant steps for a specific
  challenging reasoning step. Further, this disconnect may hinder the correct
  reasoning due to its irrelevance. To this end, we focus on improving the
  reasoning quality within each step and present BoostStep. BoostStep aligns the
  granularity between the retrieving and reasoning on step grained, and provides
  highly related ICL examples for each reasoning step with a novel `first-try'
  strategy. BoostStep provides more relevant examples than the coarse
  question-grained strategy, enhancing the model reasoning quality within each
  step steadily. BoostStep is a general and robust reasoning-enhancing method
  that not only improves standalone reasoning performance but also integrates
  seamlessly with Monte Carlo Tree Search methods (MCTS) to refine both candidate
  generation and decision-making. Quantitatively, it improves GPT-4o and
  Qwen2.5-Math-72B by 3.6\% and 2.0\% respectively on various mathematical
  benchmarks, and 7.5\% gain combined with MCTS.
paper_ai_summary: BoostStep improves large language models' reasoning quality in math problems by aligning granularity and providing relevant in-context learning examples, enhancing performance and integrating with Monte Carlo Tree Search methods.
paper_ai_keywords: large language models divide-and-conquer pipeline in-context learning granularity-mismatch negative-effect noise conquering steps question-grained step-grained reasoning quality first-try strategy Monte Carlo Tree Search GPT-4o Qwen2.5-Math-72B