title: CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code
    Completion
summary: Code completion models have made significant progress in recent years, yet
  current popular evaluation datasets, such as HumanEval and MBPP, predominantly
  focus on code completion tasks within a single file. This over-simplified
  setting falls short of representing the real-world software development
  scenario where repositories span multiple files with numerous cross-file
  dependencies, and accessing and understanding cross-file context is often
  required to complete the code correctly.
    To fill in this gap, we propose CrossCodeEval, a diverse and multilingual
  code completion benchmark that necessitates an in-depth cross-file contextual
  understanding to complete the code accurately. CrossCodeEval is built on a
  diverse set of real-world, open-sourced, permissively-licensed repositories in
  four popular programming languages: Python, Java, TypeScript, and C#. To create
  examples that strictly require cross-file context for accurate completion, we
  propose a straightforward yet efficient static-analysis-based approach to
  pinpoint the use of cross-file context within the current file.
    Extensive experiments on state-of-the-art code language models like CodeGen
  and StarCoder demonstrate that CrossCodeEval is extremely challenging when the
  relevant cross-file context is absent, and we see clear improvements when
  adding these context into the prompt. However, despite such improvements, the
  pinnacle of performance remains notably unattained even with the
  highest-performing model, indicating that CrossCodeEval is also capable of
  assessing model's capability in leveraging extensive context to make better
  code completion. Finally, we benchmarked various methods in retrieving
  cross-file context, and show that CrossCodeEval can also be used to measure the
  capability of code retrievers.
paper_authors._id: 652f4fa0f75d9cc2a4d418ad
paper_authors.name: Yangruibo Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T09:30:02.354Z
paper_authors._id: 6458648072b60ae7a381044f
paper_authors.avatarUrl: /avatars/f6e26c8663b30cf8c742e494929859ea.svg
paper_authors.fullname: Yangruibo Ding
paper_authors.type: user
paper_authors.user: yrbding
paper_authors._id: 652f4fa0f75d9cc2a4d418ae
paper_authors.name: Zijian Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T09:30:28.894Z
paper_authors._id: 6465183934acce85aea2c3b3
paper_authors.avatarUrl: /avatars/275a0811381afc980d339ba2aa1294c7.svg
paper_authors.fullname: Zijian Wang
paper_authors.type: user
paper_authors.user: zijwang
paper_authors._id: 652f4fa0f75d9cc2a4d418af
paper_authors.name: Wasi Uddin Ahmad
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T09:30:36.866Z
paper_authors._id: 6115506463c5df6b8d45fb65
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671159107866-6115506463c5df6b8d45fb65.jpeg
paper_authors.fullname: Wasi Uddin Ahmad
paper_authors.type: user
paper_authors.user: wasiahmad
paper_authors._id: 652f4fa0f75d9cc2a4d418b0
paper_authors.name: Hantian Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-18T09:31:02.495Z
paper_authors._id: 637556dafd59a439194d819d
paper_authors.avatarUrl: /avatars/a6241a3adf3bd3b1c3a6ba63933a4117.svg
paper_authors.fullname: Hantian Ding
paper_authors.type: user
paper_authors.user: dhantian
paper_authors._id: 652f4fa0f75d9cc2a4d418b1
paper_authors.name: Ming Tan
paper_authors._id: 652f4fa0f75d9cc2a4d418b2
paper_authors.name: Nihal Jain
paper_authors._id: 652f4fa0f75d9cc2a4d418b3
paper_authors.name: Murali Krishna Ramanathan
paper_authors._id: 652f4fa0f75d9cc2a4d418b4
paper_authors.name: Ramesh Nallapati
paper_authors._id: 652f4fa0f75d9cc2a4d418b5
paper_authors.name: Parminder Bhatia
paper_authors._id: 652f4fa0f75d9cc2a4d418b6
paper_authors.name: Dan Roth
paper_authors._id: 652f4fa0f75d9cc2a4d418b7
paper_authors.name: Bing Xiang
paper_title: CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code
    Completion
paper_summary: Code completion models have made significant progress in recent years, yet
  current popular evaluation datasets, such as HumanEval and MBPP, predominantly
  focus on code completion tasks within a single file. This over-simplified
  setting falls short of representing the real-world software development
  scenario where repositories span multiple files with numerous cross-file
  dependencies, and accessing and understanding cross-file context is often
  required to complete the code correctly.
    To fill in this gap, we propose CrossCodeEval, a diverse and multilingual
  code completion benchmark that necessitates an in-depth cross-file contextual
  understanding to complete the code accurately. CrossCodeEval is built on a
  diverse set of real-world, open-sourced, permissively-licensed repositories in
  four popular programming languages: Python, Java, TypeScript, and C#. To create
  examples that strictly require cross-file context for accurate completion, we
  propose a straightforward yet efficient static-analysis-based approach to
  pinpoint the use of cross-file context within the current file.
    Extensive experiments on state-of-the-art code language models like CodeGen
  and StarCoder demonstrate that CrossCodeEval is extremely challenging when the
  relevant cross-file context is absent, and we see clear improvements when
  adding these context into the prompt. However, despite such improvements, the
  pinnacle of performance remains notably unattained even with the
  highest-performing model, indicating that CrossCodeEval is also capable of
  assessing model's capability in leveraging extensive context to make better
  code completion. Finally, we benchmarked various methods in retrieving
  cross-file context, and show that CrossCodeEval can also be used to measure the
  capability of code retrievers.
paper_ai_keywords: code completion models CrossCodeEval cross-file contextual understanding permissively-licensed repositories static-analysis-based approach code language models CodeGen StarCoder cross-file context code retrievers