title: FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation
    for Feature Implementation
summary: Implementing new features in repository-level codebases is a crucial
  application of code generation models. However, current benchmarks lack a
  dedicated evaluation framework for this capability. To fill this gap, we
  introduce FEA-Bench, a benchmark designed to assess the ability of large
  language models (LLMs) to perform incremental development within code
  repositories. We collect pull requests from 83 GitHub repositories and use
  rule-based and intent-based filtering to construct task instances focused on
  new feature development. Each task instance containing code changes is paired
  with relevant unit test files to ensure that the solution can be verified. The
  feature implementation requires LLMs to simultaneously possess code completion
  capabilities for new components and code editing abilities for other relevant
  parts in the code repository, providing a more comprehensive evaluation method
  of LLMs' automated software engineering capabilities. Experimental results show
  that LLMs perform significantly worse in the FEA-Bench, highlighting
  considerable challenges in such repository-level incremental code development.
paper_authors._id: 67cf94d9f2b1fe815db6db40
paper_authors.name: Wei Li
paper_authors._id: 67cf94d9f2b1fe815db6db41
paper_authors.name: Xin Zhang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-11T01:41:47.194Z
paper_authors._id: 641a9a4b05290a135041a3ed
paper_authors.avatarUrl: /avatars/95d66ac607973abe95bd3558c6c93739.svg
paper_authors.fullname: Pluto
paper_authors.type: user
paper_authors.user: CharonBony
paper_authors._id: 67cf94d9f2b1fe815db6db42
paper_authors.name: Zhongxin Guo
paper_authors._id: 67cf94d9f2b1fe815db6db43
paper_authors.name: Shaoguang Mao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T11:04:01.624Z
paper_authors._id: 64af8d4b2cda6a37a4927d72
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/o37G6VkKSyP4N9xktFErV.png
paper_authors.fullname: Shaoguang Mao
paper_authors.type: user
paper_authors.user: dawnmsg
paper_authors._id: 67cf94d9f2b1fe815db6db44
paper_authors.name: Wen Luo
paper_authors._id: 67cf94d9f2b1fe815db6db45
paper_authors.name: Guangyue Peng
paper_authors._id: 67cf94d9f2b1fe815db6db46
paper_authors.name: Yangyu Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T11:04:18.816Z
paper_authors._id: 64c66647725ffa04b2fd6c94
paper_authors.avatarUrl: /avatars/620f63f27fa1e90423b0dc22aa8e5809.svg
paper_authors.fullname: yangyu huang
paper_authors.type: user
paper_authors.user: yangyu90
paper_authors._id: 67cf94d9f2b1fe815db6db47
paper_authors.name: Houfeng Wang
paper_authors._id: 67cf94d9f2b1fe815db6db48
paper_authors.name: Scarlett Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T11:04:34.252Z
paper_authors._id: 67366efb049bfa3a9084e8d1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Bny5bUc6qFQ_RWpEKpRAW.jpeg
paper_authors.fullname: Scarlett Li
paper_authors.type: user
paper_authors.user: lisijia0504
paper_title: FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation
    for Feature Implementation
paper_summary: Implementing new features in repository-level codebases is a crucial
  application of code generation models. However, current benchmarks lack a
  dedicated evaluation framework for this capability. To fill this gap, we
  introduce FEA-Bench, a benchmark designed to assess the ability of large
  language models (LLMs) to perform incremental development within code
  repositories. We collect pull requests from 83 GitHub repositories and use
  rule-based and intent-based filtering to construct task instances focused on
  new feature development. Each task instance containing code changes is paired
  with relevant unit test files to ensure that the solution can be verified. The
  feature implementation requires LLMs to simultaneously possess code completion
  capabilities for new components and code editing abilities for other relevant
  parts in the code repository, providing a more comprehensive evaluation method
  of LLMs' automated software engineering capabilities. Experimental results show
  that LLMs perform significantly worse in the FEA-Bench, highlighting
  considerable challenges in such repository-level incremental code development.