Use Property-Based Testing to Bridge LLM Code Generation and Validation
Large Language Models (LLMs) excel at code generation, but ensuring their
outputs to be functionally correct, especially in complex programming tasks, is
a persistent challenge. While traditional Test-Driven Development (TDD) offers
a path for code refinement, its efficacy with LLMs is often undermined by the
scarcity of high-quality test cases or the pitfalls of automated test
generation, including biased tests or inaccurate output predictions that can
misdirect the correction process. This paper introduces Property-Generated
Solver, a novel framework that leverages Property-Based Testing (PBT) to
validate high-level program properties or invariants, instead of relying on
specific input-output examples. These properties are often simpler to define
and verify than directly predicting exhaustive test oracles, breaking the
"cycle of self-deception" where tests might share flaws with the code they are
meant to validate. Property-Generated Solver employs two collaborative
LLM-based agents: a Generator dedicated to code generation and iterative
refinement, and a Tester that manages the PBT life-cycle and formulate
semantically rich feedback from property violations. The resulting
comprehensive and actionable feedback then guides the Generator in its
refinement efforts. By establishing PBT as the core validation engine within
this iterative, closed-loop paradigm, Property-Generated Solver provides a
robust mechanism for steering LLMs towards more correct and generalizable code.
Extensive experimental results on multiple code generation benchmarks
demonstrate that Property-Generated Solver achieves substantial pass@1
improvements, ranging from 23.1% to 37.3% relative gains over established TDD
methods.
685cb786696820ba1f28f286
Lehan He
685cb786696820ba1f28f287
Zeren Chen
claimed_verified
2025-06-26T09:23:31.152Z
66c44c6826efa38bc783b07a
/avatars/7a09179a2c91adc97f8db851fca37eea.svg
Zeren Chen
user
zx55
685cb786696820ba1f28f288
Zhe Zhang
685cb786696820ba1f28f289
Jing Shao
685cb786696820ba1f28f28a
Xiang Gao
685cb786696820ba1f28f28b
Lu Sheng
claimed_verified
2025-06-26T09:23:29.105Z
65b722dbe02a17f0f8d1cc6b
/avatars/65f20601ef9b8ebfdddadd737f9153d6.svg
Lu Sheng
user
lsheng2024
Use Property-Based Testing to Bridge LLM Code Generation and Validation
Large Language Models (LLMs) excel at code generation, but ensuring their
outputs to be functionally correct, especially in complex programming tasks, is
a persistent challenge. While traditional Test-Driven Development (TDD) offers
a path for code refinement, its efficacy with LLMs is often undermined by the
scarcity of high-quality test cases or the pitfalls of automated test
generation, including biased tests or inaccurate output predictions that can
misdirect the correction process. This paper introduces Property-Generated
Solver, a novel framework that leverages Property-Based Testing (PBT) to
validate high-level program properties or invariants, instead of relying on
specific input-output examples. These properties are often simpler to define
and verify than directly predicting exhaustive test oracles, breaking the
"cycle of self-deception" where tests might share flaws with the code they are
meant to validate. Property-Generated Solver employs two collaborative
LLM-based agents: a Generator dedicated to code generation and iterative
refinement, and a Tester that manages the PBT life-cycle and formulate
semantically rich feedback from property violations. The resulting
comprehensive and actionable feedback then guides the Generator in its
refinement efforts. By establishing PBT as the core validation engine within
this iterative, closed-loop paradigm, Property-Generated Solver provides a
robust mechanism for steering LLMs towards more correct and generalizable code.
Extensive experimental results on multiple code generation benchmarks
demonstrate that Property-Generated Solver achieves substantial pass@1
improvements, ranging from 23.1% to 37.3% relative gains over established TDD
methods.
A novel framework using Property-Based Testing and collaborative LLM-based agents improves code generation correctness and generalization.
Large Language Models code generation Test-Driven Development Property-Based Testing PBT iterative refinement property violations pass@1 improvements