title: DevEval: A Manually-Annotated Code Generation Benchmark Aligned with
    Real-World Code Repositories
summary: How to evaluate the coding abilities of Large Language Models (LLMs) remains
  an open question. We find that existing benchmarks are poorly aligned with
  real-world code repositories and are insufficient to evaluate the coding
  abilities of LLMs.
    To address the knowledge gap, we propose a new benchmark named DevEval, which
  has three advances. (1) DevEval aligns with real-world repositories in multiple
  dimensions, e.g., code distributions and dependency distributions. (2) DevEval
  is annotated by 13 developers and contains comprehensive annotations (e.g.,
  requirements, original repositories, reference code, and reference
  dependencies). (3) DevEval comprises 1,874 testing samples from 117
  repositories, covering 10 popular domains (e.g., Internet, Database). Based on
  DevEval, we propose repository-level code generation and evaluate 8 popular
  LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa).
  Our experiments reveal these LLMs' coding abilities in real-world code
  repositories. For example, in our experiments, the highest Pass@1 of
  gpt-4-turbo is only 53.04%. We also analyze LLMs' failed cases and summarize
  their shortcomings. We hope DevEval can facilitate the development of LLMs in
  real code repositories. DevEval, prompts, and LLMs' predictions have been
  released.
paper_authors._id: 6659eb90edd7468b910e7299
paper_authors.name: Jia Li
paper_authors._id: 6659eb90edd7468b910e729a
paper_authors.name: Ge Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:02:46.558Z
paper_authors._id: 6530b96ef68c6e013d08b413
paper_authors.avatarUrl: /avatars/ecc2874e612d0aed357d072427faf21a.svg
paper_authors.fullname: Ge Li
paper_authors.type: user
paper_authors.user: ligechina
paper_authors._id: 6659eb90edd7468b910e729b
paper_authors.name: Yunfei Zhao
paper_authors._id: 6659eb90edd7468b910e729c
paper_authors.name: Yongmin Li
paper_authors._id: 6659eb90edd7468b910e729d
paper_authors.name: Huanyu Liu
paper_authors._id: 6659eb90edd7468b910e729e
paper_authors.name: Hao Zhu
paper_authors._id: 6659eb90edd7468b910e729f
paper_authors.name: Lecheng Wang
paper_authors._id: 6659eb90edd7468b910e72a0
paper_authors.name: Kaibo Liu
paper_authors._id: 6659eb90edd7468b910e72a1
paper_authors.name: Zheng Fang
paper_authors._id: 6659eb90edd7468b910e72a2
paper_authors.name: Lanshen Wang
paper_authors._id: 6659eb90edd7468b910e72a3
paper_authors.name: Jiazheng Ding
paper_authors._id: 6659eb90edd7468b910e72a4
paper_authors.name: Xuanming Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T19:50:15.673Z
paper_authors._id: 65fc5109899083a2aad987c5
paper_authors.avatarUrl: /avatars/289dbb8128746d931118cff6f6871a45.svg
paper_authors.fullname: XUANMING ZHANG
paper_authors.type: user
paper_authors.user: XUANMINGZHANG
paper_authors._id: 6659eb90edd7468b910e72a5
paper_authors.name: Yuqi Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T19:49:57.424Z
paper_authors._id: 646d3c3aeb9268aeebc37de1
paper_authors.avatarUrl: /avatars/877aff144e43f93b390cf05f95ddf11c.svg
paper_authors.fullname: Yuqi Zhu
paper_authors.type: user
paper_authors.user: YuqiZhu29
paper_authors._id: 6659eb90edd7468b910e72a6
paper_authors.name: Yihong Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T19:49:50.485Z
paper_authors._id: 6468e631b2321e47d3277ec9
paper_authors.avatarUrl: /avatars/3acd3868232067289fad69db6c32cf87.svg
paper_authors.fullname: Yihong Dong
paper_authors.type: user
paper_authors.user: dongyh
paper_authors._id: 6659eb90edd7468b910e72a7
paper_authors.name: Zhi Jin
paper_authors._id: 6659eb90edd7468b910e72a8
paper_authors.name: Binhua Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T19:49:43.159Z
paper_authors._id: 656ddba4271c5c4e3356cf9d
paper_authors.avatarUrl: /avatars/88e47148c2dd0dc171d103d2a0e9f429.svg
paper_authors.fullname: Binhua
paper_authors.type: user
paper_authors.user: BinhuaL
paper_authors._id: 6659eb90edd7468b910e72a9
paper_authors.name: Fei Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T19:49:12.851Z
paper_authors._id: 635b8b6a37c6a2c12e2cce00
paper_authors.avatarUrl: /avatars/229fb72180529141515d1df797b33709.svg
paper_authors.fullname: Fei Huang
paper_authors.type: user
paper_authors.user: hzhwcmhf
paper_authors._id: 6659eb90edd7468b910e72aa
paper_authors.name: Yongbin Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T10:48:05.019Z
paper_authors._id: 66641b2fd8e1e34bc621e688
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66641b2fd8e1e34bc621e688/csPETwnx2zCIHSWi9uAi-.png
paper_authors.fullname: Yongbin Li
paper_authors.type: user
paper_authors.user: Yongbin-Li
paper_title: DevEval: A Manually-Annotated Code Generation Benchmark Aligned with
    Real-World Code Repositories
paper_summary: How to evaluate the coding abilities of Large Language Models (LLMs) remains
  an open question. We find that existing benchmarks are poorly aligned with
  real-world code repositories and are insufficient to evaluate the coding
  abilities of LLMs.
    To address the knowledge gap, we propose a new benchmark named DevEval, which
  has three advances. (1) DevEval aligns with real-world repositories in multiple
  dimensions, e.g., code distributions and dependency distributions. (2) DevEval
  is annotated by 13 developers and contains comprehensive annotations (e.g.,
  requirements, original repositories, reference code, and reference
  dependencies). (3) DevEval comprises 1,874 testing samples from 117
  repositories, covering 10 popular domains (e.g., Internet, Database). Based on
  DevEval, we propose repository-level code generation and evaluate 8 popular
  LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa).
  Our experiments reveal these LLMs' coding abilities in real-world code
  repositories. For example, in our experiments, the highest Pass@1 of
  gpt-4-turbo is only 53.04%. We also analyze LLMs' failed cases and summarize
  their shortcomings. We hope DevEval can facilitate the development of LLMs in
  real code repositories. DevEval, prompts, and LLMs' predictions have been
  released.
paper_ai_keywords: repository-level code generation