FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation
  for Feature Implementation
Implementing new features in repository-level codebases is a crucial
application of code generation models. However, current benchmarks lack a
dedicated evaluation framework for this capability. To fill this gap, we
introduce FEA-Bench, a benchmark designed to assess the ability of large
language models (LLMs) to perform incremental development within code
repositories. We collect pull requests from 83 GitHub repositories and use
rule-based and intent-based filtering to construct task instances focused on
new feature development. Each task instance containing code changes is paired
with relevant unit test files to ensure that the solution can be verified. The
feature implementation requires LLMs to simultaneously possess code completion
capabilities for new components and code editing abilities for other relevant
parts in the code repository, providing a more comprehensive evaluation method
of LLMs' automated software engineering capabilities. Experimental results show
that LLMs perform significantly worse in the FEA-Bench, highlighting
considerable challenges in such repository-level incremental code development.
67cf94d9f2b1fe815db6db40
Wei Li
67cf94d9f2b1fe815db6db41
Xin Zhang
extracted_pending
2025-03-11T01:41:47.194Z
641a9a4b05290a135041a3ed
/avatars/95d66ac607973abe95bd3558c6c93739.svg
Pluto
user
CharonBony
67cf94d9f2b1fe815db6db42
Zhongxin Guo
67cf94d9f2b1fe815db6db43
Shaoguang Mao
admin_assigned
2025-03-11T11:04:01.624Z
64af8d4b2cda6a37a4927d72
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/o37G6VkKSyP4N9xktFErV.png
Shaoguang Mao
user
dawnmsg
67cf94d9f2b1fe815db6db44
Wen Luo
67cf94d9f2b1fe815db6db45
Guangyue Peng
67cf94d9f2b1fe815db6db46
Yangyu Huang
admin_assigned
2025-03-11T11:04:18.816Z
64c66647725ffa04b2fd6c94
/avatars/620f63f27fa1e90423b0dc22aa8e5809.svg
yangyu huang
user
yangyu90
67cf94d9f2b1fe815db6db47
Houfeng Wang
67cf94d9f2b1fe815db6db48
Scarlett Li
admin_assigned
2025-03-11T11:04:34.252Z
67366efb049bfa3a9084e8d1
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Bny5bUc6qFQ_RWpEKpRAW.jpeg
Scarlett Li
user
lisijia0504
FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation
  for Feature Implementation
Implementing new features in repository-level codebases is a crucial
application of code generation models. However, current benchmarks lack a
dedicated evaluation framework for this capability. To fill this gap, we
introduce FEA-Bench, a benchmark designed to assess the ability of large
language models (LLMs) to perform incremental development within code
repositories. We collect pull requests from 83 GitHub repositories and use
rule-based and intent-based filtering to construct task instances focused on
new feature development. Each task instance containing code changes is paired
with relevant unit test files to ensure that the solution can be verified. The
feature implementation requires LLMs to simultaneously possess code completion
capabilities for new components and code editing abilities for other relevant
parts in the code repository, providing a more comprehensive evaluation method
of LLMs' automated software engineering capabilities. Experimental results show
that LLMs perform significantly worse in the FEA-Bench, highlighting
considerable challenges in such repository-level incremental code development.
A new benchmark, FEA-Bench, evaluates large language models' incremental coding capabilities in software repositories, showing their limitations.
FSA-Bench large language models repository-level code development incremental development code completion code editing automated software engineering