title: AdaptiveStep: Automatically Dividing Reasoning Step through Model
    Confidence
summary: Current approaches for training Process Reward Models (PRMs) often involve
  breaking down responses into multiple reasoning steps using rule-based
  techniques, such as using predefined placeholder tokens or setting the
  reasoning step's length into a fixed size. These approaches overlook the fact
  that specific words do not typically mark true decision points in a text. To
  address this, we propose AdaptiveStep, a method that divides reasoning steps
  based on the model's confidence in predicting the next word. This division
  method provides more decision-making information at each step, enhancing
  downstream tasks, such as reward model learning. Moreover, our method does not
  require manual annotation. We demonstrate its effectiveness through experiments
  with AdaptiveStep-trained PRMs in mathematical reasoning and code generation
  tasks. Experimental results indicate that the outcome PRM achieves
  state-of-the-art Best-of-N performance, surpassing greedy search strategy with
  token-level value-guided decoding, while also reducing construction costs by
  over 30% compared to existing open-source PRMs. In addition, we provide a
  thorough analysis and case study on the PRM's performance, transferability, and
  generalization capabilities.
paper_authors._id: 67b6a9a7c721bee91cac2888
paper_authors.name: Yuliang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T17:11:40.282Z
paper_authors._id: 6529f79e802e3d1a4f8ec662
paper_authors.avatarUrl: /avatars/d05320c370a6497d8792ef5acb563dd5.svg
paper_authors.fullname: Yuliang Liu
paper_authors.type: user
paper_authors.user: yuliang03181
paper_authors._id: 67b6a9a7c721bee91cac2889
paper_authors.name: Junjie Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T13:05:14.854Z
paper_authors._id: 660e1bb6e089e44d2906a7e8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/660e1bb6e089e44d2906a7e8/MJU8nHQ0xgiPzO8J6L8EC.jpeg
paper_authors.fullname: Junjie Lu
paper_authors.type: user
paper_authors.user: Lux0926
paper_authors._id: 67b6a9a7c721bee91cac288a
paper_authors.name: Zhaoling Chen
paper_authors._id: 67b6a9a7c721bee91cac288b
paper_authors.name: Chaofeng Qu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T17:11:10.302Z
paper_authors._id: 65d0ae74c4d2b2e40281256e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65d0ae74c4d2b2e40281256e/nNTJkAkHsu9Fru4CpRphC.jpeg
paper_authors.fullname: Chaofeng QU
paper_authors.type: user
paper_authors.user: kylebrovloski
paper_authors._id: 67b6a9a7c721bee91cac288c
paper_authors.name: Jason Klein Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T17:11:16.565Z
paper_authors._id: 6731c04d5f55903a1d8c307c
paper_authors.avatarUrl: /avatars/704b1d628c55e3141194b08736f21267.svg
paper_authors.fullname: Jason Klein Liu
paper_authors.type: user
paper_authors.user: jasonkleinlove
paper_authors._id: 67b6a9a7c721bee91cac288d
paper_authors.name: Chonghan Liu
paper_authors._id: 67b6a9a7c721bee91cac288e
paper_authors.name: Zefan Cai
paper_authors._id: 67b6a9a7c721bee91cac288f
paper_authors.name: Yunhui Xia
paper_authors._id: 67b6a9a7c721bee91cac2890
paper_authors.name: Li Zhao
paper_authors._id: 67b6a9a7c721bee91cac2891
paper_authors.name: Jiang Bian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T17:00:08.549Z
paper_authors._id: 63f253f8f4e30ffd2bd308fb
paper_authors.avatarUrl: /avatars/303f4c7ee588f638acf78a7966786e1e.svg
paper_authors.fullname: Jiang Bian
paper_authors.type: user
paper_authors.user: bianjiang
paper_authors._id: 67b6a9a7c721bee91cac2892
paper_authors.name: Chuheng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T16:59:49.312Z
paper_authors._id: 64c32727c370f29a10334d35
paper_authors.avatarUrl: /avatars/a3284bdd2e51f437433b89a96a904448.svg
paper_authors.fullname: ZhangChuheng
paper_authors.type: user
paper_authors.user: zhangchuheng123
paper_authors._id: 67b6a9a7c721bee91cac2893
paper_authors.name: Wei Shen
paper_authors._id: 67b6a9a7c721bee91cac2894
paper_authors.name: Zhouhan Lin
paper_title: AdaptiveStep: Automatically Dividing Reasoning Step through Model
    Confidence
paper_summary: Current approaches for training Process Reward Models (PRMs) often involve
  breaking down responses into multiple reasoning steps using rule-based
  techniques, such as using predefined placeholder tokens or setting the
  reasoning step's length into a fixed size. These approaches overlook the fact
  that specific words do not typically mark true decision points in a text. To
  address this, we propose AdaptiveStep, a method that divides reasoning steps
  based on the model's confidence in predicting the next word. This division
  method provides more decision-making information at each step, enhancing
  downstream tasks, such as reward model learning. Moreover, our method does not
  require manual annotation. We demonstrate its effectiveness through experiments
  with AdaptiveStep-trained PRMs in mathematical reasoning and code generation
  tasks. Experimental results indicate that the outcome PRM achieves
  state-of-the-art Best-of-N performance, surpassing greedy search strategy with
  token-level value-guided decoding, while also reducing construction costs by
  over 30% compared to existing open-source PRMs. In addition, we provide a
  thorough analysis and case study on the PRM's performance, transferability, and
  generalization capabilities.
paper_ai_summary: AdaptiveStep improves reward model training by dynamically segmenting reasoning steps based on model confidence, enhancing performance in mathematical reasoning and code generation without manual annotation.
paper_ai_keywords: Process Reward Models AdaptiveStep rule-based techniques placeholder tokens fixed size decision points model confidence downstream tasks reward model learning mathematical reasoning code generation Best-of-N performance greedy search strategy token-level value-guided decoding construction costs