Rethinking Verification for LLM Code Generation: From Generation to
  Testing
Large language models (LLMs) have recently achieved notable success in
code-generation benchmarks such as HumanEval and LiveCodeBench. However, a
detailed examination reveals that these evaluation suites often comprise only a
limited number of homogeneous test cases, resulting in subtle faults going
undetected. This not only artificially inflates measured performance but also
compromises accurate reward estimation in reinforcement learning frameworks
utilizing verifiable rewards (RLVR). To address these critical shortcomings, we
systematically investigate the test-case generation (TCG) task by proposing
multi-dimensional metrics designed to rigorously quantify test-suite
thoroughness. Furthermore, we introduce a human-LLM collaborative method
(SAGA), leveraging human programming expertise with LLM reasoning capability,
aimed at significantly enhancing both the coverage and the quality of generated
test cases. In addition, we develop a TCGBench to facilitate the study of the
TCG task. Experiments show that SAGA achieves a detection rate of 90.62% and a
verifier accuracy of 32.58% on TCGBench. The Verifier Accuracy (Verifier Acc)
of the code generation evaluation benchmark synthesized by SAGA is 10.78%
higher than that of LiveCodeBench-v6. These results demonstrate the
effectiveness of our proposed method. We hope this work contributes to building
a scalable foundation for reliable LLM code evaluation, further advancing RLVR
in code generation, and paving the way for automated adversarial test synthesis
and adaptive benchmark integration.
686f1da1d938c25d68441b1b
Zihan Ma
claimed_verified
2025-07-10T07:09:33.413Z
677e869467f3bb8d8215eec6
https://cdn-avatars.huggingface.co/v1/production/uploads/677e869467f3bb8d8215eec6/kEC6JOKObgLHA22jRcP4H.jpeg
Zihan Ma
user
MichaelErchi
686f1da1d938c25d68441b1c
Taolin Zhang
claimed_verified
2025-07-10T13:14:31.682Z
64c8835a4cc48498134364be
/avatars/a26120216483c6bf3b830c0cf9c4008e.svg
TaolinZhang
user
iridescentttt
686f1da1d938c25d68441b1d
Maosong Cao
686f1da1d938c25d68441b1e
Wenwei Zhang
686f1da1d938c25d68441b1f
Minnan Luo
686f1da1d938c25d68441b20
Songyang Zhang
claimed_verified
2025-07-10T13:14:29.511Z
630716d11801ecc7d2595021
/avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
Songyang Zhang
user
zsytony
686f1da1d938c25d68441b21
Kai Chen
Rethinking Verification for LLM Code Generation: From Generation to
  Testing
Large language models (LLMs) have recently achieved notable success in
code-generation benchmarks such as HumanEval and LiveCodeBench. However, a
detailed examination reveals that these evaluation suites often comprise only a
limited number of homogeneous test cases, resulting in subtle faults going
undetected. This not only artificially inflates measured performance but also
compromises accurate reward estimation in reinforcement learning frameworks
utilizing verifiable rewards (RLVR). To address these critical shortcomings, we
systematically investigate the test-case generation (TCG) task by proposing
multi-dimensional metrics designed to rigorously quantify test-suite
thoroughness. Furthermore, we introduce a human-LLM collaborative method
(SAGA), leveraging human programming expertise with LLM reasoning capability,
aimed at significantly enhancing both the coverage and the quality of generated
test cases. In addition, we develop a TCGBench to facilitate the study of the
TCG task. Experiments show that SAGA achieves a detection rate of 90.62% and a
verifier accuracy of 32.58% on TCGBench. The Verifier Accuracy (Verifier Acc)
of the code generation evaluation benchmark synthesized by SAGA is 10.78%
higher than that of LiveCodeBench-v6. These results demonstrate the
effectiveness of our proposed method. We hope this work contributes to building
a scalable foundation for reliable LLM code evaluation, further advancing RLVR
in code generation, and paving the way for automated adversarial test synthesis
and adaptive benchmark integration.
A collaborative method combining human expertise and LLM reasoning enhances test-case generation for code evaluation, improving detection rates and verifier accuracy in benchmarks.
large language models code-generation HumanEval LiveCodeBench test-case generation multi-dimensional metrics human-LLM collaboration SAGA TCGBench reinforcement learning frameworks verifiable rewards RLVR automated adversarial test synthesis adaptive benchmark integration