title: Benchmarking AI Models in Software Engineering: A Review, Search Tool,
    and Enhancement Protocol
summary: Benchmarks are essential for consistent evaluation and reproducibility. The
  integration of Artificial Intelligence into Software Engineering (AI4SE) has
  given rise to numerous benchmarks for tasks such as code generation and bug
  fixing. However, this surge presents challenges: (1) scattered benchmark
  knowledge across tasks, (2) difficulty in selecting relevant benchmarks, (3)
  the absence of a uniform standard for benchmark development, and (4)
  limitations of existing benchmarks. In this paper, we review 173 studies and
  identify 204 AI4SE benchmarks. We classify these benchmarks, analyze their
  limitations, and expose gaps in practices. Based on our review, we created
  BenchScout, a semantic search tool to find relevant benchmarks, using automated
  clustering of the contexts from associated studies. We conducted a user study
  with 22 participants to evaluate BenchScout's usability, effectiveness, and
  intuitiveness which resulted in average scores of 4.5, 4.0, and 4.1 out of 5.
  To advance benchmarking standards, we propose BenchFrame, a unified method to
  enhance benchmark quality. As a case study, we applied BenchFrame to the
  HumanEval benchmark and addressed its main limitations. This led to
  HumanEvalNext, featuring (1) corrected errors, (2) improved language
  conversion, (3) expanded test coverage, and (4) increased difficulty. We then
  evaluated ten state-of-the-art code language models on HumanEval,
  HumanEvalPlus, and HumanEvalNext. On HumanEvalNext, models showed a pass@1
  score reduction of 31.22% and 19.94% compared to HumanEval and HumanEvalPlus,
  respectively.
paper_authors._id: 67d0a239967ead9b5aff9883
paper_authors.name: Roham Koohestani
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:41:54.151Z
paper_authors._id: 655a627aab0644b531a02eb1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/9rW6X1idfx1p5omky67D6.jpeg
paper_authors.fullname: Roham Koohestani
paper_authors.type: user
paper_authors.user: RohamKoohestani
paper_authors._id: 67d0a239967ead9b5aff9884
paper_authors.name: Philippe de Bekker
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-11T20:51:06.669Z
paper_authors._id: 655213d1968a2554a5e8212a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/3XM_b9imWk-pwoueJwAZB.jpeg
paper_authors.fullname: Philippe de Bekker
paper_authors.type: user
paper_authors.user: philippedebekker
paper_authors._id: 67d0a239967ead9b5aff9885
paper_authors.name: Maliheh Izadi
paper_title: Benchmarking AI Models in Software Engineering: A Review, Search Tool,
    and Enhancement Protocol
paper_summary: Benchmarks are essential for consistent evaluation and reproducibility. The
  integration of Artificial Intelligence into Software Engineering (AI4SE) has
  given rise to numerous benchmarks for tasks such as code generation and bug
  fixing. However, this surge presents challenges: (1) scattered benchmark
  knowledge across tasks, (2) difficulty in selecting relevant benchmarks, (3)
  the absence of a uniform standard for benchmark development, and (4)
  limitations of existing benchmarks. In this paper, we review 173 studies and
  identify 204 AI4SE benchmarks. We classify these benchmarks, analyze their
  limitations, and expose gaps in practices. Based on our review, we created
  BenchScout, a semantic search tool to find relevant benchmarks, using automated
  clustering of the contexts from associated studies. We conducted a user study
  with 22 participants to evaluate BenchScout's usability, effectiveness, and
  intuitiveness which resulted in average scores of 4.5, 4.0, and 4.1 out of 5.
  To advance benchmarking standards, we propose BenchFrame, a unified method to
  enhance benchmark quality. As a case study, we applied BenchFrame to the
  HumanEval benchmark and addressed its main limitations. This led to
  HumanEvalNext, featuring (1) corrected errors, (2) improved language
  conversion, (3) expanded test coverage, and (4) increased difficulty. We then
  evaluated ten state-of-the-art code language models on HumanEval,
  HumanEvalPlus, and HumanEvalNext. On HumanEvalNext, models showed a pass@1
  score reduction of 31.22% and 19.94% compared to HumanEval and HumanEvalPlus,
  respectively.
paper_ai_summary: BenchScout and BenchFrame improve the selection and quality of AI4SE benchmarks, addressing multiple limitations in existing benchmarking practices.
paper_ai_keywords: Artificial Intelligence Software Engineering AI4SE code generation bug fixing benchmarks clustering semantic search usability effectiveness intuitiveness HumanEval HumanEvalPlus HumanEvalNext pass@1 score