Dynamic Scaling of Unit Tests for Code Reward Modeling
Current large language models (LLMs) often struggle to produce accurate
responses on the first attempt for complex reasoning tasks like code
generation. Prior research tackles this challenge by generating multiple
candidate solutions and validating them with LLM-generated unit tests. The
execution results of unit tests serve as reward signals to identify correct
solutions. As LLMs always confidently make mistakes, these unit tests are not
reliable, thereby diminishing the quality of reward signals. Motivated by the
observation that scaling the number of solutions improves LLM performance, we
explore the impact of scaling unit tests to enhance reward signal quality. Our
pioneer experiment reveals a positive correlation between the number of unit
tests and reward signal quality, with greater benefits observed in more
challenging problems. Based on these insights, we propose CodeRM-8B, a
lightweight yet effective unit test generator that enables efficient and
high-quality unit test scaling. Additionally, we implement a dynamic scaling
mechanism that adapts the number of unit tests based on problem difficulty,
further improving efficiency. Experimental results show that our approach
significantly improves performance across various models on three benchmarks
(e.g., with gains of 18.43% for Llama3-8B and 3.42% for GPT-4o-mini on
HumanEval Plus).
67774afb5c6bccc41be7ba19
Zeyao Ma
extracted_pending
2025-01-03T02:27:07.916Z
6384c07fdfffab4824ff45fb
https://cdn-avatars.huggingface.co/v1/production/uploads/1669644372381-noauth.jpeg
Zeyao Ma
user
KAKA22
67774afb5c6bccc41be7ba1a
Xiaokang Zhang
67774afb5c6bccc41be7ba1b
Jing Zhang
67774afb5c6bccc41be7ba1c
Jifan Yu
67774afb5c6bccc41be7ba1d
Sijia Luo
67774afb5c6bccc41be7ba1e
Jie Tang
Dynamic Scaling of Unit Tests for Code Reward Modeling
Current large language models (LLMs) often struggle to produce accurate
responses on the first attempt for complex reasoning tasks like code
generation. Prior research tackles this challenge by generating multiple
candidate solutions and validating them with LLM-generated unit tests. The
execution results of unit tests serve as reward signals to identify correct
solutions. As LLMs always confidently make mistakes, these unit tests are not
reliable, thereby diminishing the quality of reward signals. Motivated by the
observation that scaling the number of solutions improves LLM performance, we
explore the impact of scaling unit tests to enhance reward signal quality. Our
pioneer experiment reveals a positive correlation between the number of unit
tests and reward signal quality, with greater benefits observed in more
challenging problems. Based on these insights, we propose CodeRM-8B, a
lightweight yet effective unit test generator that enables efficient and
high-quality unit test scaling. Additionally, we implement a dynamic scaling
mechanism that adapts the number of unit tests based on problem difficulty,
further improving efficiency. Experimental results show that our approach
significantly improves performance across various models on three benchmarks
(e.g., with gains of 18.43% for Llama3-8B and 3.42% for GPT-4o-mini on
HumanEval Plus).
Generating and dynamically scaling unit tests improves the performance of large language models on complex reasoning tasks like code generation.
large language models complex reasoning tasks code generation unit tests reward signals problem difficulty CodeRM-8B dynamic scaling mechanism HumanEval Plus Llama3-8B GPT-4o-mini