On the Impacts of Contexts on Repository-Level Code Generation
<|reference_start|>On the Impacts of Contexts on Repository-Level Code Generation: CodeLLMs have gained widespread adoption for code generation tasks, yet their capacity to handle repository-level code generation with complex contextual dependencies remains underexplored. Our work underscores the critical importance of leveraging repository-level contexts to generate executable and functionally correct code. We present \textbf{\methodnamews}, a novel benchmark designed to evaluate repository-level code generation, with a focus on three key aspects: executability, functional correctness through comprehensive test case generation, and accurate utilization of cross-file contexts. Our study examines a controlled scenario where developers specify essential code dependencies (contexts), challenging models to integrate them effectively. Additionally, we introduce an instruction-tuned dataset that enhances CodeLLMs' ability to leverage dependencies, along with a new metric, \textit{Dependency Invocation Rate (DIR)}, to quantify context utilization. Experimental results reveal that while pretrained LLMs demonstrate superior performance in terms of correctness, instruction-tuned models excel in context utilization and debugging capabilities. \methodnamews offers a comprehensive evaluation framework for assessing code functionality and alignment with developer intent, thereby advancing the development of more reliable CodeLLMs for real-world applications. The dataset and source code are available at~\url{https://github.com/FSoft-AI4Code/RepoExec}.<|reference_end|>
@article{hai2024on,
    title={On the Impacts of Contexts on Repository-Level Code Generation},
    author={Nam Le Hai, Dung Manh Nguyen, Nghi D. Q. Bui},
    journal={arXiv preprint arXiv:2406.11927},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.11927},
    primaryClass={cs.SE cs.AI}
}