AdaptiveStep: Automatically Dividing Reasoning Step through Model
  Confidence
Current approaches for training Process Reward Models (PRMs) often involve
breaking down responses into multiple reasoning steps using rule-based
techniques, such as using predefined placeholder tokens or setting the
reasoning step's length into a fixed size. These approaches overlook the fact
that specific words do not typically mark true decision points in a text. To
address this, we propose AdaptiveStep, a method that divides reasoning steps
based on the model's confidence in predicting the next word. This division
method provides more decision-making information at each step, enhancing
downstream tasks, such as reward model learning. Moreover, our method does not
require manual annotation. We demonstrate its effectiveness through experiments
with AdaptiveStep-trained PRMs in mathematical reasoning and code generation
tasks. Experimental results indicate that the outcome PRM achieves
state-of-the-art Best-of-N performance, surpassing greedy search strategy with
token-level value-guided decoding, while also reducing construction costs by
over 30% compared to existing open-source PRMs. In addition, we provide a
thorough analysis and case study on the PRM's performance, transferability, and
generalization capabilities.
67b6a9a7c721bee91cac2888
Yuliang Liu
admin_assigned
2025-02-20T17:11:40.282Z
6529f79e802e3d1a4f8ec662
/avatars/d05320c370a6497d8792ef5acb563dd5.svg
Yuliang Liu
user
yuliang03181
67b6a9a7c721bee91cac2889
Junjie Lu
claimed_verified
2025-02-24T13:05:14.854Z
660e1bb6e089e44d2906a7e8
https://cdn-avatars.huggingface.co/v1/production/uploads/660e1bb6e089e44d2906a7e8/MJU8nHQ0xgiPzO8J6L8EC.jpeg
Junjie Lu
user
Lux0926
67b6a9a7c721bee91cac288a
Zhaoling Chen
67b6a9a7c721bee91cac288b
Chaofeng Qu
admin_assigned
2025-02-20T17:11:10.302Z
65d0ae74c4d2b2e40281256e
https://cdn-avatars.huggingface.co/v1/production/uploads/65d0ae74c4d2b2e40281256e/nNTJkAkHsu9Fru4CpRphC.jpeg
Chaofeng QU
user
kylebrovloski
67b6a9a7c721bee91cac288c
Jason Klein Liu
admin_assigned
2025-02-20T17:11:16.565Z
6731c04d5f55903a1d8c307c
/avatars/704b1d628c55e3141194b08736f21267.svg
Jason Klein Liu
user
jasonkleinlove
67b6a9a7c721bee91cac288d
Chonghan Liu
67b6a9a7c721bee91cac288e
Zefan Cai
67b6a9a7c721bee91cac288f
Yunhui Xia
67b6a9a7c721bee91cac2890
Li Zhao
67b6a9a7c721bee91cac2891
Jiang Bian
admin_assigned
2025-02-20T17:00:08.549Z
63f253f8f4e30ffd2bd308fb
/avatars/303f4c7ee588f638acf78a7966786e1e.svg
Jiang Bian
user
bianjiang
67b6a9a7c721bee91cac2892
Chuheng Zhang
admin_assigned
2025-02-20T16:59:49.312Z
64c32727c370f29a10334d35
/avatars/a3284bdd2e51f437433b89a96a904448.svg
ZhangChuheng
user
zhangchuheng123
67b6a9a7c721bee91cac2893
Wei Shen
67b6a9a7c721bee91cac2894
Zhouhan Lin
AdaptiveStep: Automatically Dividing Reasoning Step through Model
  Confidence
Current approaches for training Process Reward Models (PRMs) often involve
breaking down responses into multiple reasoning steps using rule-based
techniques, such as using predefined placeholder tokens or setting the
reasoning step's length into a fixed size. These approaches overlook the fact
that specific words do not typically mark true decision points in a text. To
address this, we propose AdaptiveStep, a method that divides reasoning steps
based on the model's confidence in predicting the next word. This division
method provides more decision-making information at each step, enhancing
downstream tasks, such as reward model learning. Moreover, our method does not
require manual annotation. We demonstrate its effectiveness through experiments
with AdaptiveStep-trained PRMs in mathematical reasoning and code generation
tasks. Experimental results indicate that the outcome PRM achieves
state-of-the-art Best-of-N performance, surpassing greedy search strategy with
token-level value-guided decoding, while also reducing construction costs by
over 30% compared to existing open-source PRMs. In addition, we provide a
thorough analysis and case study on the PRM's performance, transferability, and
generalization capabilities.
AdaptiveStep improves reward model training by dynamically segmenting reasoning steps based on model confidence, enhancing performance in mathematical reasoning and code generation without manual annotation.
Process Reward Models AdaptiveStep rule-based techniques placeholder tokens fixed size decision points model confidence downstream tasks reward model learning mathematical reasoning code generation Best-of-N performance greedy search strategy token-level value-guided decoding construction costs