title: M2rc-Eval: Massively Multilingual Repository-level Code Completion
    Evaluation
summary: Repository-level code completion has drawn great attention in software
  engineering, and several benchmark datasets have been introduced. However,
  existing repository-level code completion benchmarks usually focus on a limited
  number of languages (<5), which cannot evaluate the general code intelligence
  abilities across different languages for existing code Large Language Models
  (LLMs). Besides, the existing benchmarks usually report overall average scores
  of different languages, where the fine-grained abilities in different
  completion scenarios are ignored. Therefore, to facilitate the research of code
  LLMs in multilingual scenarios, we propose a massively multilingual
  repository-level code completion benchmark covering 18 programming languages
  (called M2RC-EVAL), and two types of fine-grained annotations (i.e.,
  bucket-level and semantic-level) on different completion scenarios are
  provided, where we obtain these annotations based on the parsed abstract syntax
  tree. Moreover, we also curate a massively multilingual instruction corpora
  M2RC- INSTRUCT dataset to improve the repository-level code completion
  abilities of existing code LLMs. Comprehensive experimental results demonstrate
  the effectiveness of our M2RC-EVAL and M2RC-INSTRUCT.
paper_authors._id: 672840ed89116e24a4296489
paper_authors.name: Jiaheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T21:30:50.375Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 672840ed89116e24a429648a
paper_authors.name: Ken Deng
paper_authors._id: 672840ed89116e24a429648b
paper_authors.name: Congnan Liu
paper_authors._id: 672840ed89116e24a429648c
paper_authors.name: Jian Yang
paper_authors._id: 672840ed89116e24a429648d
paper_authors.name: Shukai Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T21:31:17.337Z
paper_authors._id: 666914ba38d9327ca72134c4
paper_authors.avatarUrl: /avatars/e65e2e27fc01064909ba257565387d10.svg
paper_authors.fullname: Shukai Liu
paper_authors.type: user
paper_authors.user: skLiu
paper_authors._id: 672840ed89116e24a429648e
paper_authors.name: He Zhu
paper_authors._id: 672840ed89116e24a429648f
paper_authors.name: Peng Zhao
paper_authors._id: 672840ed89116e24a4296490
paper_authors.name: Linzheng Chai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T21:31:28.233Z
paper_authors._id: 64ba096e760936217a3ad2e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ba096e760936217a3ad2e2/aNQK83Jg5PsBkY0UDg-RA.jpeg
paper_authors.fullname: Linzheng Chai
paper_authors.type: user
paper_authors.user: Challenging666
paper_authors._id: 672840ed89116e24a4296491
paper_authors.name: Yanan Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T21:31:41.476Z
paper_authors._id: 65a33a99db5c00652eaa5e4c
paper_authors.avatarUrl: /avatars/23346ff87dfd2db2463c11137de9fbf2.svg
paper_authors.fullname: Yanan Wu
paper_authors.type: user
paper_authors.user: wuyanan513
paper_authors._id: 672840ed89116e24a4296492
paper_authors.name: Ke Jin
paper_authors._id: 672840ed89116e24a4296493
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T21:32:54.677Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 672840ed89116e24a4296494
paper_authors.name: Zekun Wang
paper_authors._id: 672840ed89116e24a4296495
paper_authors.name: Guoan Zhang
paper_authors._id: 672840ed89116e24a4296496
paper_authors.name: Bangyu Xiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T21:31:55.129Z
paper_authors._id: 646f40e4a6a58aa295064bff
paper_authors.avatarUrl: /avatars/8e31117e4f49e6404d139592cfb2bff1.svg
paper_authors.fullname: XIANG BANGYU
paper_authors.type: user
paper_authors.user: BANGYU
paper_authors._id: 672840ed89116e24a4296497
paper_authors.name: Wenbo Su
paper_authors._id: 672840ed89116e24a4296498
paper_authors.name: Bo Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-04T21:32:13.499Z
paper_authors._id: 62c695ad5aae1c624ca992e2
paper_authors.avatarUrl: /avatars/20d10fb3338e4bd4dc59e88a18cb2617.svg
paper_authors.fullname: Bo Zheng
paper_authors.type: user
paper_authors.user: bzheng
paper_title: M2rc-Eval: Massively Multilingual Repository-level Code Completion
    Evaluation
paper_summary: Repository-level code completion has drawn great attention in software
  engineering, and several benchmark datasets have been introduced. However,
  existing repository-level code completion benchmarks usually focus on a limited
  number of languages (<5), which cannot evaluate the general code intelligence
  abilities across different languages for existing code Large Language Models
  (LLMs). Besides, the existing benchmarks usually report overall average scores
  of different languages, where the fine-grained abilities in different
  completion scenarios are ignored. Therefore, to facilitate the research of code
  LLMs in multilingual scenarios, we propose a massively multilingual
  repository-level code completion benchmark covering 18 programming languages
  (called M2RC-EVAL), and two types of fine-grained annotations (i.e.,
  bucket-level and semantic-level) on different completion scenarios are
  provided, where we obtain these annotations based on the parsed abstract syntax
  tree. Moreover, we also curate a massively multilingual instruction corpora
  M2RC- INSTRUCT dataset to improve the repository-level code completion
  abilities of existing code LLMs. Comprehensive experimental results demonstrate
  the effectiveness of our M2RC-EVAL and M2RC-INSTRUCT.
paper_ai_keywords: large language models (LLMs) repository-level code completion benchmark datasets data intelligence abilities general code intelligence multilingual scenarios massively multilingual repository-level code completion benchmark M2RC-EVAL fine-grained annotations bucket-level annotations semantic-level annotations abstract syntax tree (AST) massively multilingual instruction corpora M2RC-INSTRUCT repository-level code completion abilities