Outcome-Refining Process Supervision for Code Generation
Large Language Models have demonstrated remarkable capabilities in code
generation, yet they often struggle with complex programming tasks that require
deep algorithmic reasoning. While process supervision through learned reward
models shows promise in guiding reasoning steps, it requires expensive training
data and suffers from unreliable evaluation. We propose Outcome-Refining
Process Supervision, a novel paradigm that treats outcome refinement itself as
the process to be supervised. Our framework leverages concrete execution
signals to ground the supervision of reasoning steps, while using
tree-structured exploration to maintain multiple solution trajectories
simultaneously. Experiments demonstrate that our approach enables even smaller
models to achieve high success accuracy and performance metrics on competitive
programming tasks, creates more reliable verification than traditional reward
models without requiring training PRMs. Our approach achieves significant
improvements across 5 models and 3 datasets: an average of 26.9% increase in
correctness and 42.2% in efficiency. The results suggest that providing
structured reasoning space with concrete verification signals is crucial for
solving complex programming tasks. We open-source all our code and data at:
https://github.com/zhuohaoyu/ORPS
676a444d9315022860ac1f0e
Zhuohao Yu
claimed_verified
2024-12-30T19:34:53.843Z
61e24808b31e7cc38eb84d37
/avatars/65fbea940fad211462ecc5ad725e0c28.svg
Zhuohao Yu
user
zhuohaoyu
676a444d9315022860ac1f0f
Weizheng Gu
676a444d9315022860ac1f10
Yidong Wang
676a444d9315022860ac1f11
Zhengran Zeng
676a444d9315022860ac1f12
Jindong Wang
claimed_verified
2025-06-02T07:54:33.006Z
6204cc0d522e40b4a18d86e2
/avatars/18daf2de5671e711dc745388dd60569d.svg
Jindong Wang
user
jindongwang
676a444d9315022860ac1f13
Wei Ye
676a444d9315022860ac1f14
Shikun Zhang
Outcome-Refining Process Supervision for Code Generation
Large Language Models have demonstrated remarkable capabilities in code
generation, yet they often struggle with complex programming tasks that require
deep algorithmic reasoning. While process supervision through learned reward
models shows promise in guiding reasoning steps, it requires expensive training
data and suffers from unreliable evaluation. We propose Outcome-Refining
Process Supervision, a novel paradigm that treats outcome refinement itself as
the process to be supervised. Our framework leverages concrete execution
signals to ground the supervision of reasoning steps, while using
tree-structured exploration to maintain multiple solution trajectories
simultaneously. Experiments demonstrate that our approach enables even smaller
models to achieve high success accuracy and performance metrics on competitive
programming tasks, creates more reliable verification than traditional reward
models without requiring training PRMs. Our approach achieves significant
improvements across 5 models and 3 datasets: an average of 26.9% increase in
correctness and 42.2% in efficiency. The results suggest that providing
structured reasoning space with concrete verification signals is crucial for
solving complex programming tasks. We open-source all our code and data at:
https://github.com/zhuohaoyu/ORPS
Outcome-Refining Process Supervision uses concrete execution signals and tree-structured exploration to enhance algorithmic reasoning in code generation tasks.
outcome refinement process supervision learned reward models concrete execution signals tree-structured exploration algorithmic reasoning code generation competitive programming tasks verification