Can LLMs Generate High-Quality Test Cases for Algorithm Problems?
  TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure
We introduce TestCase-Eval, a new benchmark for systematic evaluation of LLMs
in test-case generation. TestCase-Eval includes 500 algorithm problems and
100,000 human-crafted solutions from the Codeforces platform. It focuses on two
pivotal tasks: (1) Fault Coverage, which measures how well LLM-generated test
sets probe diverse input scenarios and cover a wide range of potential failure
modes. (2) Fault Exposure, which evaluates whether LLMs can craft a tailored
test input that reveals a specific incorrect code implementation. We provide a
comprehensive assessment of 19 state-of-the-art open-source and proprietary
LLMs on TestCase-Eval, offering insights into their strengths and limitations
in generating effective test cases for algorithm problems.
685234030164cd1316710502
Zheyuan Yang
685234030164cd1316710503
Zexi Kuang
685234030164cd1316710504
Xue Xia
685234030164cd1316710505
Yilun Zhao
claimed_verified
2025-06-18T10:59:08.920Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
user
yilunzhao
Can LLMs Generate High-Quality Test Cases for Algorithm Problems?
  TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure
We introduce TestCase-Eval, a new benchmark for systematic evaluation of LLMs
in test-case generation. TestCase-Eval includes 500 algorithm problems and
100,000 human-crafted solutions from the Codeforces platform. It focuses on two
pivotal tasks: (1) Fault Coverage, which measures how well LLM-generated test
sets probe diverse input scenarios and cover a wide range of potential failure
modes. (2) Fault Exposure, which evaluates whether LLMs can craft a tailored
test input that reveals a specific incorrect code implementation. We provide a
comprehensive assessment of 19 state-of-the-art open-source and proprietary
LLMs on TestCase-Eval, offering insights into their strengths and limitations
in generating effective test cases for algorithm problems.
TestCase-Eval is a benchmark for evaluating LLMs in generating comprehensive and targeted test cases for algorithm problems.
test-case generation Fault Coverage Fault Exposure LLMs algorithm problems human-crafted solutions Codeforces test sets failure modes incorrect code implementation