Benchmarking AI Models in Software Engineering: A Review, Search Tool,
  and Enhancement Protocol
Benchmarks are essential for consistent evaluation and reproducibility. The
integration of Artificial Intelligence into Software Engineering (AI4SE) has
given rise to numerous benchmarks for tasks such as code generation and bug
fixing. However, this surge presents challenges: (1) scattered benchmark
knowledge across tasks, (2) difficulty in selecting relevant benchmarks, (3)
the absence of a uniform standard for benchmark development, and (4)
limitations of existing benchmarks. In this paper, we review 173 studies and
identify 204 AI4SE benchmarks. We classify these benchmarks, analyze their
limitations, and expose gaps in practices. Based on our review, we created
BenchScout, a semantic search tool to find relevant benchmarks, using automated
clustering of the contexts from associated studies. We conducted a user study
with 22 participants to evaluate BenchScout's usability, effectiveness, and
intuitiveness which resulted in average scores of 4.5, 4.0, and 4.1 out of 5.
To advance benchmarking standards, we propose BenchFrame, a unified method to
enhance benchmark quality. As a case study, we applied BenchFrame to the
HumanEval benchmark and addressed its main limitations. This led to
HumanEvalNext, featuring (1) corrected errors, (2) improved language
conversion, (3) expanded test coverage, and (4) increased difficulty. We then
evaluated ten state-of-the-art code language models on HumanEval,
HumanEvalPlus, and HumanEvalNext. On HumanEvalNext, models showed a pass@1
score reduction of 31.22% and 19.94% compared to HumanEval and HumanEvalPlus,
respectively.
67d0a239967ead9b5aff9883
Roham Koohestani
claimed_verified
2025-03-12T08:41:54.151Z
655a627aab0644b531a02eb1
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/9rW6X1idfx1p5omky67D6.jpeg
Roham Koohestani
user
RohamKoohestani
67d0a239967ead9b5aff9884
Philippe de Bekker
extracted_confirmed
2025-06-08T13:49:56.900Z
655213d1968a2554a5e8212a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/3XM_b9imWk-pwoueJwAZB.jpeg
Philippe de Bekker
user
philippedebekker
67d0a239967ead9b5aff9885
Maliheh Izadi
Benchmarking AI Models in Software Engineering: A Review, Search Tool,
  and Enhancement Protocol
Benchmarks are essential for consistent evaluation and reproducibility. The
integration of Artificial Intelligence into Software Engineering (AI4SE) has
given rise to numerous benchmarks for tasks such as code generation and bug
fixing. However, this surge presents challenges: (1) scattered benchmark
knowledge across tasks, (2) difficulty in selecting relevant benchmarks, (3)
the absence of a uniform standard for benchmark development, and (4)
limitations of existing benchmarks. In this paper, we review 173 studies and
identify 204 AI4SE benchmarks. We classify these benchmarks, analyze their
limitations, and expose gaps in practices. Based on our review, we created
BenchScout, a semantic search tool to find relevant benchmarks, using automated
clustering of the contexts from associated studies. We conducted a user study
with 22 participants to evaluate BenchScout's usability, effectiveness, and
intuitiveness which resulted in average scores of 4.5, 4.0, and 4.1 out of 5.
To advance benchmarking standards, we propose BenchFrame, a unified method to
enhance benchmark quality. As a case study, we applied BenchFrame to the
HumanEval benchmark and addressed its main limitations. This led to
HumanEvalNext, featuring (1) corrected errors, (2) improved language
conversion, (3) expanded test coverage, and (4) increased difficulty. We then
evaluated ten state-of-the-art code language models on HumanEval,
HumanEvalPlus, and HumanEvalNext. On HumanEvalNext, models showed a pass@1
score reduction of 31.22% and 19.94% compared to HumanEval and HumanEvalPlus,
respectively.
BenchScout and BenchFrame improve the selection and quality of AI4SE benchmarks, addressing multiple limitations in existing benchmarking practices.
Artificial Intelligence Software Engineering AI4SE code generation bug fixing benchmarks clustering semantic search usability effectiveness intuitiveness HumanEval HumanEvalPlus HumanEvalNext pass@1 score