McEval: Massively Multilingual Code Evaluation
Code large language models (LLMs) have shown remarkable advances in code
understanding, completion, and generation tasks. Programming benchmarks,
comprised of a selection of code challenges and corresponding test cases, serve
as a standard to evaluate the capability of different LLMs in such tasks.
However, most existing benchmarks primarily focus on Python and are still
restricted to a limited number of languages, where other languages are
translated from the Python samples (e.g. MultiPL-E) degrading the data
diversity. To further facilitate the research of code LLMs, we propose a
massively multilingual code benchmark covering 40 programming languages
(McEval) with 16K test samples, which substantially pushes the limits of code
LLMs in multilingual scenarios. The benchmark contains challenging code
completion, understanding, and generation evaluation tasks with finely curated
massively multilingual instruction corpora McEval-Instruct. In addition, we
introduce an effective multilingual coder mCoder trained on McEval-Instruct to
support multilingual programming language generation. Extensive experimental
results on McEval show that there is still a difficult journey between
open-source models and closed-source LLMs (e.g. GPT-series models) in numerous
languages. The instruction corpora, evaluation benchmark, and leaderboard are
available at https://mceval.github.io/.
666913357acc6e95a98c8012
Linzheng Chai
claimed_verified
2024-06-12T07:29:09.991Z
64ba096e760936217a3ad2e2
https://cdn-avatars.huggingface.co/v1/production/uploads/64ba096e760936217a3ad2e2/aNQK83Jg5PsBkY0UDg-RA.jpeg
Linzheng Chai
user
Challenging666
666913357acc6e95a98c8013
Shukai Liu
claimed_verified
2024-06-13T09:32:05.183Z
666914ba38d9327ca72134c4
/avatars/e65e2e27fc01064909ba257565387d10.svg
Shukai Liu
user
skLiu
666913357acc6e95a98c8014
Jian Yang
claimed_verified
2024-10-14T19:14:40.146Z
64ccb9bfead94891d12aef42
/avatars/c54809d43d93d3f0766bd2555cecc4e3.svg
Yang Jian
user
CSJianYang
666913357acc6e95a98c8015
Yuwei Yin
claimed_verified
2024-06-12T07:29:19.814Z
64510a21f800611f94f0d9f8
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/lOeHK9Bvt3IXcB7Urx6jZ.jpeg
Yuwei Yin
user
yuweiyin
666913357acc6e95a98c8016
Ke Jin
666913357acc6e95a98c8017
Jiaheng Liu
admin_assigned
2024-06-14T15:02:50.941Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
666913357acc6e95a98c8018
Tao Sun
666913357acc6e95a98c8019
Ge Zhang
claimed_verified
2024-06-12T07:29:04.533Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
666913357acc6e95a98c801a
Changyu Ren
666913357acc6e95a98c801b
Hongcheng Guo
admin_assigned
2024-06-14T15:07:23.804Z
62728f4f6253fe2068da1021
https://cdn-avatars.huggingface.co/v1/production/uploads/62728f4f6253fe2068da1021/KZ65X0EH98AF3zXemPiap.jpeg
Hongcheng Gao
user
HongchengGao
666913357acc6e95a98c801c
Zekun Wang
claimed_verified
2024-10-07T07:31:34.204Z
6149a9e95347647e6bb68882
https://cdn-avatars.huggingface.co/v1/production/uploads/6149a9e95347647e6bb68882/Jddln1FxScCeVgTSCNBpr.png
Zekun Moore Wang
user
ZenMoore
666913357acc6e95a98c801d
Boyang Wang
666913357acc6e95a98c801e
Xianjie Wu
666913357acc6e95a98c801f
Bing Wang
666913357acc6e95a98c8020
Tongliang Li
admin_assigned
2024-06-14T15:09:03.329Z
663860b8d37124438e73fc13
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VCrseZtFRCfcm7iIZP1SP.jpeg
Tongliang Li
user
tonyliangli
666913357acc6e95a98c8021
Liqun Yang
666913357acc6e95a98c8022
Sufeng Duan
666913357acc6e95a98c8023
Zhoujun Li
McEval: Massively Multilingual Code Evaluation
Code large language models (LLMs) have shown remarkable advances in code
understanding, completion, and generation tasks. Programming benchmarks,
comprised of a selection of code challenges and corresponding test cases, serve
as a standard to evaluate the capability of different LLMs in such tasks.
However, most existing benchmarks primarily focus on Python and are still
restricted to a limited number of languages, where other languages are
translated from the Python samples (e.g. MultiPL-E) degrading the data
diversity. To further facilitate the research of code LLMs, we propose a
massively multilingual code benchmark covering 40 programming languages
(McEval) with 16K test samples, which substantially pushes the limits of code
LLMs in multilingual scenarios. The benchmark contains challenging code
completion, understanding, and generation evaluation tasks with finely curated
massively multilingual instruction corpora McEval-Instruct. In addition, we
introduce an effective multilingual coder mCoder trained on McEval-Instruct to
support multilingual programming language generation. Extensive experimental
results on McEval show that there is still a difficult journey between
open-source models and closed-source LLMs (e.g. GPT-series models) in numerous
languages. The instruction corpora, evaluation benchmark, and leaderboard are
available at https://mceval.github.io/.
A new multilingual benchmark McEval with 16K samples across 40 languages evaluates and facilitates the development of multilingual code large language models.
code large language models code understanding code completion code generation programming benchmarks test cases multilingual code benchmark code completion code understanding instruction corpora multilingual coder mCoder