M2rc-Eval: Massively Multilingual Repository-level Code Completion
  Evaluation
Repository-level code completion has drawn great attention in software
engineering, and several benchmark datasets have been introduced. However,
existing repository-level code completion benchmarks usually focus on a limited
number of languages (<5), which cannot evaluate the general code intelligence
abilities across different languages for existing code Large Language Models
(LLMs). Besides, the existing benchmarks usually report overall average scores
of different languages, where the fine-grained abilities in different
completion scenarios are ignored. Therefore, to facilitate the research of code
LLMs in multilingual scenarios, we propose a massively multilingual
repository-level code completion benchmark covering 18 programming languages
(called M2RC-EVAL), and two types of fine-grained annotations (i.e.,
bucket-level and semantic-level) on different completion scenarios are
provided, where we obtain these annotations based on the parsed abstract syntax
tree. Moreover, we also curate a massively multilingual instruction corpora
M2RC- INSTRUCT dataset to improve the repository-level code completion
abilities of existing code LLMs. Comprehensive experimental results demonstrate
the effectiveness of our M2RC-EVAL and M2RC-INSTRUCT.
672840ed89116e24a4296489
Jiaheng Liu
admin_assigned
2024-11-04T21:30:50.375Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
672840ed89116e24a429648a
Ken Deng
672840ed89116e24a429648b
Congnan Liu
672840ed89116e24a429648c
Jian Yang
672840ed89116e24a429648d
Shukai Liu
admin_assigned
2024-11-04T21:31:17.337Z
666914ba38d9327ca72134c4
/avatars/e65e2e27fc01064909ba257565387d10.svg
Shukai Liu
user
skLiu
672840ed89116e24a429648e
He Zhu
672840ed89116e24a429648f
Peng Zhao
672840ed89116e24a4296490
Linzheng Chai
admin_assigned
2024-11-04T21:31:28.233Z
64ba096e760936217a3ad2e2
https://cdn-avatars.huggingface.co/v1/production/uploads/64ba096e760936217a3ad2e2/aNQK83Jg5PsBkY0UDg-RA.jpeg
Linzheng Chai
user
Challenging666
672840ed89116e24a4296491
Yanan Wu
admin_assigned
2024-11-04T21:31:41.476Z
65a33a99db5c00652eaa5e4c
/avatars/23346ff87dfd2db2463c11137de9fbf2.svg
Yanan Wu
user
wuyanan513
672840ed89116e24a4296492
Ke Jin
672840ed89116e24a4296493
Ge Zhang
admin_assigned
2024-11-04T21:32:54.677Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
672840ed89116e24a4296494
Zekun Wang
672840ed89116e24a4296495
Guoan Zhang
672840ed89116e24a4296496
Bangyu Xiang
admin_assigned
2024-11-04T21:31:55.129Z
646f40e4a6a58aa295064bff
/avatars/8e31117e4f49e6404d139592cfb2bff1.svg
XIANG BANGYU
user
BANGYU
672840ed89116e24a4296497
Wenbo Su
672840ed89116e24a4296498
Bo Zheng
admin_assigned
2024-11-04T21:32:13.499Z
62c695ad5aae1c624ca992e2
/avatars/20d10fb3338e4bd4dc59e88a18cb2617.svg
Bo Zheng
user
bzheng
M2rc-Eval: Massively Multilingual Repository-level Code Completion
  Evaluation
Repository-level code completion has drawn great attention in software
engineering, and several benchmark datasets have been introduced. However,
existing repository-level code completion benchmarks usually focus on a limited
number of languages (<5), which cannot evaluate the general code intelligence
abilities across different languages for existing code Large Language Models
(LLMs). Besides, the existing benchmarks usually report overall average scores
of different languages, where the fine-grained abilities in different
completion scenarios are ignored. Therefore, to facilitate the research of code
LLMs in multilingual scenarios, we propose a massively multilingual
repository-level code completion benchmark covering 18 programming languages
(called M2RC-EVAL), and two types of fine-grained annotations (i.e.,
bucket-level and semantic-level) on different completion scenarios are
provided, where we obtain these annotations based on the parsed abstract syntax
tree. Moreover, we also curate a massively multilingual instruction corpora
M2RC- INSTRUCT dataset to improve the repository-level code completion
abilities of existing code LLMs. Comprehensive experimental results demonstrate
the effectiveness of our M2RC-EVAL and M2RC-INSTRUCT.
A new multilingual benchmark and instruction dataset are introduced to evaluate and improve repository-level code completion for Large Language Models across 18 programming languages.
repository-level code completion Large Language Models benchmark datasets multilingual scenarios massively multilingual bucket-level annotations semantic-level annotations abstract syntax tree code intelligence