LLM-Powered Test Case Generation for Detecting Tricky Bugs
<|reference_start|>LLM-Powered Test Case Generation for Detecting Tricky Bugs: Conventional automated test generation tools struggle to generate test oracles and tricky bug-revealing test inputs. Large Language Models (LLMs) can be prompted to produce test inputs and oracles for a program directly, but the precision of the tests can be very low for complex scenarios (only 6.3% based on our experiments). To fill this gap, this paper proposes AID, which combines LLMs with differential testing to generate fault-revealing test inputs and oracles targeting plausibly correct programs (i.e., programs that have passed all the existing tests). In particular, AID selects test inputs that yield diverse outputs on a set of program variants generated by LLMs, then constructs the test oracle based on the outputs. We evaluate AID on two large-scale datasets with tricky bugs: TrickyBugs and EvalPlus, and compare it with three state-of-the-art baselines. The evaluation results show that the recall, precision, and F1 score of AID outperform the state-of-the-art by up to 1.80x, 2.65x, and 1.66x, respectively.<|reference_end|>
@article{liu2024llm-powered,
    title={LLM-Powered Test Case Generation for Detecting Tricky Bugs},
    author={Kaibo Liu and Yiyang Liu and Zhenpeng Chen and Jie M. Zhang and Yudong
  Han and Yun Ma and Ge Li and Gang Huang},
    journal={arXiv preprint arXiv:2404.10304},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.10304},
    primaryClass={cs.SE cs.LG}
}