Learning to Generate Unit Tests for Automated Debugging
Unit tests (UTs) play an instrumental role in assessing code correctness as
well as providing feedback to a large language model (LLM) as it iteratively
debugs faulty code, motivating automated test generation. However, we uncover a
trade-off between generating unit test inputs that reveal errors when given a
faulty code and correctly predicting the unit test output without access to the
gold solution. To address this trade-off, we propose UTGen, which teaches LLMs
to generate unit test inputs that reveal errors along with their correct
expected outputs based on task descriptions and candidate code. We integrate
UTGen into UTDebug, a robust debugging pipeline that uses generated tests to
help LLMs debug effectively. Since model-generated tests can provide noisy
signals (e.g., from incorrectly predicted outputs), UTDebug (i) scales UTGen
via test-time compute to improve UT output prediction, and (ii) validates and
back-tracks edits based on multiple generated UTs to avoid overfitting. We show
that UTGen outperforms UT generation baselines by 7.59% based on a metric
measuring the presence of both error-revealing UT inputs and correct UT
outputs. When used with UTDebug, we find that feedback from UTGen's unit tests
improves pass@1 accuracy of Qwen-2.5 7B on HumanEvalFix and our own harder
debugging split of MBPP+ by over 3% and 12.35% (respectively) over other
LLM-based UT generation baselines.
67a280644fdf4d9187507d74
Archiki Prasad
admin_assigned
2025-02-04T21:11:22.710Z
607aeae5d2cd8c150e6ae074
/avatars/a087743b98b6fe2181283a9610db4ec4.svg
Archiki Prasad
user
archiki
67a280644fdf4d9187507d75
Elias Stengel-Eskin
admin_assigned
2025-02-04T21:11:36.824Z
61781c4caf41befe8ff060e8
/avatars/8871d7b046fc28cbc8638228da8e9737.svg
Elias Stengel-Eskin
user
esteng
67a280644fdf4d9187507d76
Justin Chih-Yao Chen
67a280644fdf4d9187507d77
Zaid Khan
claimed_verified
2025-02-26T08:39:28.111Z
6301c3e0a123c93a5fb295ff
https://cdn-avatars.huggingface.co/v1/production/uploads/1661060051926-noauth.jpeg
Zaid Khan
user
codezakh
67a280644fdf4d9187507d78
Mohit Bansal
admin_assigned
2025-02-04T21:12:17.347Z
665d9d3a057f7c508f98c625
https://cdn-avatars.huggingface.co/v1/production/uploads/665d9d3a057f7c508f98c625/u1R9P9sJoAl4zEIcetbPy.jpeg
Mohit Bansal
user
mohitbansal
Learning to Generate Unit Tests for Automated Debugging
Unit tests (UTs) play an instrumental role in assessing code correctness as
well as providing feedback to a large language model (LLM) as it iteratively
debugs faulty code, motivating automated test generation. However, we uncover a
trade-off between generating unit test inputs that reveal errors when given a
faulty code and correctly predicting the unit test output without access to the
gold solution. To address this trade-off, we propose UTGen, which teaches LLMs
to generate unit test inputs that reveal errors along with their correct
expected outputs based on task descriptions and candidate code. We integrate
UTGen into UTDebug, a robust debugging pipeline that uses generated tests to
help LLMs debug effectively. Since model-generated tests can provide noisy
signals (e.g., from incorrectly predicted outputs), UTDebug (i) scales UTGen
via test-time compute to improve UT output prediction, and (ii) validates and
back-tracks edits based on multiple generated UTs to avoid overfitting. We show
that UTGen outperforms UT generation baselines by 7.59% based on a metric
measuring the presence of both error-revealing UT inputs and correct UT
outputs. When used with UTDebug, we find that feedback from UTGen's unit tests
improves pass@1 accuracy of Qwen-2.5 7B on HumanEvalFix and our own harder
debugging split of MBPP+ by over 3% and 12.35% (respectively) over other
LLM-based UT generation baselines.
UTGen generates error-revealing unit tests with correct outputs for LLMs, improving debugging efficiency through integration with UTDebug.
LLM unit tests UTGen UTDebug task descriptions candidate code error-revealing inputs noisy signals validation back-tracking pass@1 accuracy HumanEvalFix debugging split MBPP+