title: REPOEXEC: Evaluate Code Generation with a Repository-Level Executable
    Benchmark
summary: The ability of CodeLLMs to generate executable and functionally correct code
  at the repository-level scale remains largely unexplored. We introduce
  RepoExec, a novel benchmark for evaluating code generation at the
  repository-level scale. RepoExec focuses on three main aspects: executability,
  functional correctness through automated test case generation with high
  coverage rate, and carefully crafted cross-file contexts to accurately generate
  code. Our work explores a controlled scenario where developers specify
  necessary code dependencies, challenging the model to integrate these
  accurately. Experiments show that while pretrained LLMs outperform
  instruction-tuned models in correctness, the latter excel in utilizing provided
  dependencies and demonstrating debugging capabilities. We also introduce a new
  instruction-tuned dataset that focuses on code dependencies and demonstrate
  that CodeLLMs fine-tuned on our dataset have a better capability to leverage
  these dependencies effectively. RepoExec aims to provide a comprehensive
  evaluation of code functionality and alignment with developer intent, paving
  the way for more reliable and applicable CodeLLMs in real-world scenarios. The
  dataset and source code can be found
  at~https://github.com/FSoft-AI4Code/RepoExec.
paper_authors._id: 6674fc013ae9e9c65996d66e
paper_authors.name: Nam Le Hai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T08:17:37.057Z
paper_authors._id: 63a27e21f30c4642278de3ea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a27e21f30c4642278de3ea/zj0mGDv9DEK9JYHFjfaZq.jpeg
paper_authors.fullname: Le Hai Nam
paper_authors.type: user
paper_authors.user: NamCyan
paper_authors._id: 6674fc013ae9e9c65996d66f
paper_authors.name: Dung Manh Nguyen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T11:48:53.306Z
paper_authors._id: 6177978d1e3e9ecbf77ddeef
paper_authors.avatarUrl: /avatars/607e28f09ee8995c9a13579ad46f25ec.svg
paper_authors.fullname: Dung Manh Nguyen
paper_authors.type: user
paper_authors.user: nmd2000
paper_authors._id: 6674fc013ae9e9c65996d670
paper_authors.name: Nghi D. Q. Bui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T11:49:46.436Z
paper_authors._id: 63a44141de134926a2ba0458
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a44141de134926a2ba0458/FP3uxpsKL1nBrrKyiFne3.jpeg
paper_authors.fullname: Nghi Bui
paper_authors.type: user
paper_authors.user: bdqnghi
paper_title: REPOEXEC: Evaluate Code Generation with a Repository-Level Executable
    Benchmark
paper_summary: The ability of CodeLLMs to generate executable and functionally correct code
  at the repository-level scale remains largely unexplored. We introduce
  RepoExec, a novel benchmark for evaluating code generation at the
  repository-level scale. RepoExec focuses on three main aspects: executability,
  functional correctness through automated test case generation with high
  coverage rate, and carefully crafted cross-file contexts to accurately generate
  code. Our work explores a controlled scenario where developers specify
  necessary code dependencies, challenging the model to integrate these
  accurately. Experiments show that while pretrained LLMs outperform
  instruction-tuned models in correctness, the latter excel in utilizing provided
  dependencies and demonstrating debugging capabilities. We also introduce a new
  instruction-tuned dataset that focuses on code dependencies and demonstrate
  that CodeLLMs fine-tuned on our dataset have a better capability to leverage
  these dependencies effectively. RepoExec aims to provide a comprehensive
  evaluation of code functionality and alignment with developer intent, paving
  the way for more reliable and applicable CodeLLMs in real-world scenarios. The
  dataset and source code can be found
  at~https://github.com/FSoft-AI4Code/RepoExec.
paper_ai_summary: RepoExec is a benchmark for evaluating repository-level code generation focusing on executability, functional correctness, and dependency integration.
paper_ai_keywords: LLMs CodeLLMs RepoExec executable functional correctness automated test case generation cross-file contexts code dependencies fine-tuned dataset debugging capabilities