title: Learning to Generate Unit Tests for Automated Debugging
summary: Unit tests (UTs) play an instrumental role in assessing code correctness as
  well as providing feedback to a large language model (LLM) as it iteratively
  debugs faulty code, motivating automated test generation. However, we uncover a
  trade-off between generating unit test inputs that reveal errors when given a
  faulty code and correctly predicting the unit test output without access to the
  gold solution. To address this trade-off, we propose UTGen, which teaches LLMs
  to generate unit test inputs that reveal errors along with their correct
  expected outputs based on task descriptions and candidate code. We integrate
  UTGen into UTDebug, a robust debugging pipeline that uses generated tests to
  help LLMs debug effectively. Since model-generated tests can provide noisy
  signals (e.g., from incorrectly predicted outputs), UTDebug (i) scales UTGen
  via test-time compute to improve UT output prediction, and (ii) validates and
  back-tracks edits based on multiple generated UTs to avoid overfitting. We show
  that UTGen outperforms UT generation baselines by 7.59% based on a metric
  measuring the presence of both error-revealing UT inputs and correct UT
  outputs. When used with UTDebug, we find that feedback from UTGen's unit tests
  improves pass@1 accuracy of Qwen-2.5 7B on HumanEvalFix and our own harder
  debugging split of MBPP+ by over 3% and 12.35% (respectively) over other
  LLM-based UT generation baselines.
paper_authors._id: 67a280644fdf4d9187507d74
paper_authors.name: Archiki Prasad
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T21:11:22.710Z
paper_authors._id: 607aeae5d2cd8c150e6ae074
paper_authors.avatarUrl: /avatars/a087743b98b6fe2181283a9610db4ec4.svg
paper_authors.fullname: Archiki Prasad
paper_authors.type: user
paper_authors.user: archiki
paper_authors._id: 67a280644fdf4d9187507d75
paper_authors.name: Elias Stengel-Eskin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T21:11:36.824Z
paper_authors._id: 61781c4caf41befe8ff060e8
paper_authors.avatarUrl: /avatars/8871d7b046fc28cbc8638228da8e9737.svg
paper_authors.fullname: Elias Stengel-Eskin
paper_authors.type: user
paper_authors.user: esteng
paper_authors._id: 67a280644fdf4d9187507d76
paper_authors.name: Justin Chih-Yao Chen
paper_authors._id: 67a280644fdf4d9187507d77
paper_authors.name: Zaid Khan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-26T08:39:28.111Z
paper_authors._id: 6301c3e0a123c93a5fb295ff
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661060051926-noauth.jpeg
paper_authors.fullname: Zaid Khan
paper_authors.type: user
paper_authors.user: codezakh
paper_authors._id: 67a280644fdf4d9187507d78
paper_authors.name: Mohit Bansal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T21:12:17.347Z
paper_authors._id: 665d9d3a057f7c508f98c625
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/665d9d3a057f7c508f98c625/u1R9P9sJoAl4zEIcetbPy.jpeg
paper_authors.fullname: Mohit Bansal
paper_authors.type: user
paper_authors.user: mohitbansal
paper_title: Learning to Generate Unit Tests for Automated Debugging
paper_summary: Unit tests (UTs) play an instrumental role in assessing code correctness as
  well as providing feedback to a large language model (LLM) as it iteratively
  debugs faulty code, motivating automated test generation. However, we uncover a
  trade-off between generating unit test inputs that reveal errors when given a
  faulty code and correctly predicting the unit test output without access to the
  gold solution. To address this trade-off, we propose UTGen, which teaches LLMs
  to generate unit test inputs that reveal errors along with their correct
  expected outputs based on task descriptions and candidate code. We integrate
  UTGen into UTDebug, a robust debugging pipeline that uses generated tests to
  help LLMs debug effectively. Since model-generated tests can provide noisy
  signals (e.g., from incorrectly predicted outputs), UTDebug (i) scales UTGen
  via test-time compute to improve UT output prediction, and (ii) validates and
  back-tracks edits based on multiple generated UTs to avoid overfitting. We show
  that UTGen outperforms UT generation baselines by 7.59% based on a metric
  measuring the presence of both error-revealing UT inputs and correct UT
  outputs. When used with UTDebug, we find that feedback from UTGen's unit tests
  improves pass@1 accuracy of Qwen-2.5 7B on HumanEvalFix and our own harder
  debugging split of MBPP+ by over 3% and 12.35% (respectively) over other
  LLM-based UT generation baselines.