title: Can Language Models Replace Programmers? REPOCOD Says 'Not Yet'
summary: Large language models (LLMs) have shown remarkable ability in code generation
  with more than 90 pass@1 in solving Python coding problems in HumanEval and
  MBPP. Such high accuracy leads to the question: can LLMs replace human
  programmers? Existing manual crafted, simple, or single-line code generation
  benchmarks cannot answer this question due to their gap with real-world
  software development. To answer this question, we propose REPOCOD, a code
  generation benchmark with 980 problems collected from 11 popular real-world
  projects, with more than 58% of them requiring file-level or repository-level
  context information. In addition, REPOCOD has the longest average canonical
  solution length (331.6 tokens) and the highest average cyclomatic complexity
  (9.00) compared to existing benchmarks. In our evaluations on ten LLMs, none of
  the models can achieve more than 30 pass@1 on REPOCOD, disclosing the necessity
  of building stronger LLMs that can help developers in real-world software
  development.
paper_authors._id: 672282a1125cc4e9e2715f76
paper_authors.name: Shanchao Liang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-30T19:01:54.565Z
paper_authors._id: 6604b3c18191f5a7efa86eba
paper_authors.avatarUrl: /avatars/1ee0ae7e999f1fb3b068297eabb1bc8f.svg
paper_authors.fullname: Shanchao Liang
paper_authors.type: user
paper_authors.user: shanchao
paper_authors._id: 672282a1125cc4e9e2715f77
paper_authors.name: Yiran Hu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-30T21:09:13.540Z
paper_authors._id: 658bb78f2e1f5bf05c2a3da9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/658bb78f2e1f5bf05c2a3da9/c4keEyiRRGXww0kZQdQph.jpeg
paper_authors.fullname: Yiran Hu
paper_authors.type: user
paper_authors.user: Yiran-Hu1007
paper_authors._id: 672282a1125cc4e9e2715f78
paper_authors.name: Nan Jiang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-30T20:31:21.269Z
paper_authors._id: 629e4ca2f2bda18349b6d330
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/629e4ca2f2bda18349b6d330/gSnGTLm2ugpINECylbuuQ.jpeg
paper_authors.fullname: Nan Jiang
paper_authors.type: user
paper_authors.user: jiang719
paper_authors._id: 672282a1125cc4e9e2715f79
paper_authors.name: Lin Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-31T09:54:53.523Z
paper_authors._id: 65831342f9c5cda913df366a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65831342f9c5cda913df366a/h7MLYt--shRYQj4-q5XmR.jpeg
paper_authors.fullname: Lin Tan
paper_authors.type: user
paper_authors.user: lin-tan
paper_title: Can Language Models Replace Programmers? REPOCOD Says 'Not Yet'
paper_summary: Large language models (LLMs) have shown remarkable ability in code generation
  with more than 90 pass@1 in solving Python coding problems in HumanEval and
  MBPP. Such high accuracy leads to the question: can LLMs replace human
  programmers? Existing manual crafted, simple, or single-line code generation
  benchmarks cannot answer this question due to their gap with real-world
  software development. To answer this question, we propose REPOCOD, a code
  generation benchmark with 980 problems collected from 11 popular real-world
  projects, with more than 58% of them requiring file-level or repository-level
  context information. In addition, REPOCOD has the longest average canonical
  solution length (331.6 tokens) and the highest average cyclomatic complexity
  (9.00) compared to existing benchmarks. In our evaluations on ten LLMs, none of
  the models can achieve more than 30 pass@1 on REPOCOD, disclosing the necessity
  of building stronger LLMs that can help developers in real-world software
  development.