title: McEval: Massively Multilingual Code Evaluation
summary: Code large language models (LLMs) have shown remarkable advances in code
  understanding, completion, and generation tasks. Programming benchmarks,
  comprised of a selection of code challenges and corresponding test cases, serve
  as a standard to evaluate the capability of different LLMs in such tasks.
  However, most existing benchmarks primarily focus on Python and are still
  restricted to a limited number of languages, where other languages are
  translated from the Python samples (e.g. MultiPL-E) degrading the data
  diversity. To further facilitate the research of code LLMs, we propose a
  massively multilingual code benchmark covering 40 programming languages
  (McEval) with 16K test samples, which substantially pushes the limits of code
  LLMs in multilingual scenarios. The benchmark contains challenging code
  completion, understanding, and generation evaluation tasks with finely curated
  massively multilingual instruction corpora McEval-Instruct. In addition, we
  introduce an effective multilingual coder mCoder trained on McEval-Instruct to
  support multilingual programming language generation. Extensive experimental
  results on McEval show that there is still a difficult journey between
  open-source models and closed-source LLMs (e.g. GPT-series models) in numerous
  languages. The instruction corpora, evaluation benchmark, and leaderboard are
  available at https://mceval.github.io/.
paper_authors._id: 666913357acc6e95a98c8012
paper_authors.name: Linzheng Chai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-12T07:29:09.991Z
paper_authors._id: 64ba096e760936217a3ad2e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ba096e760936217a3ad2e2/aNQK83Jg5PsBkY0UDg-RA.jpeg
paper_authors.fullname: Linzheng Chai
paper_authors.type: user
paper_authors.user: Challenging666
paper_authors._id: 666913357acc6e95a98c8013
paper_authors.name: Shukai Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:32:05.183Z
paper_authors._id: 666914ba38d9327ca72134c4
paper_authors.avatarUrl: /avatars/e65e2e27fc01064909ba257565387d10.svg
paper_authors.fullname: Shukai Liu
paper_authors.type: user
paper_authors.user: skLiu
paper_authors._id: 666913357acc6e95a98c8014
paper_authors.name: Jian Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T19:14:40.146Z
paper_authors._id: 64ccb9bfead94891d12aef42
paper_authors.avatarUrl: /avatars/c54809d43d93d3f0766bd2555cecc4e3.svg
paper_authors.fullname: Yang Jian
paper_authors.type: user
paper_authors.user: CSJianYang
paper_authors._id: 666913357acc6e95a98c8015
paper_authors.name: Yuwei Yin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-12T07:29:19.814Z
paper_authors._id: 64510a21f800611f94f0d9f8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/lOeHK9Bvt3IXcB7Urx6jZ.jpeg
paper_authors.fullname: Yuwei Yin
paper_authors.type: user
paper_authors.user: yuweiyin
paper_authors._id: 666913357acc6e95a98c8016
paper_authors.name: Ke Jin
paper_authors._id: 666913357acc6e95a98c8017
paper_authors.name: Jiaheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T15:02:50.941Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 666913357acc6e95a98c8018
paper_authors.name: Tao Sun
paper_authors._id: 666913357acc6e95a98c8019
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-12T07:29:04.533Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 666913357acc6e95a98c801a
paper_authors.name: Changyu Ren
paper_authors._id: 666913357acc6e95a98c801b
paper_authors.name: Hongcheng Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T15:07:23.804Z
paper_authors._id: 62728f4f6253fe2068da1021
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62728f4f6253fe2068da1021/KZ65X0EH98AF3zXemPiap.jpeg
paper_authors.fullname: Hongcheng Gao
paper_authors.type: user
paper_authors.user: HongchengGao
paper_authors._id: 666913357acc6e95a98c801c
paper_authors.name: Zekun Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-07T07:31:34.204Z
paper_authors._id: 6149a9e95347647e6bb68882
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6149a9e95347647e6bb68882/Jddln1FxScCeVgTSCNBpr.png
paper_authors.fullname: Zekun Moore Wang
paper_authors.type: user
paper_authors.user: ZenMoore
paper_authors._id: 666913357acc6e95a98c801d
paper_authors.name: Boyang Wang
paper_authors._id: 666913357acc6e95a98c801e
paper_authors.name: Xianjie Wu
paper_authors._id: 666913357acc6e95a98c801f
paper_authors.name: Bing Wang
paper_authors._id: 666913357acc6e95a98c8020
paper_authors.name: Tongliang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T15:09:03.329Z
paper_authors._id: 663860b8d37124438e73fc13
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VCrseZtFRCfcm7iIZP1SP.jpeg
paper_authors.fullname: Tongliang Li
paper_authors.type: user
paper_authors.user: tonyliangli
paper_authors._id: 666913357acc6e95a98c8021
paper_authors.name: Liqun Yang
paper_authors._id: 666913357acc6e95a98c8022
paper_authors.name: Sufeng Duan
paper_authors._id: 666913357acc6e95a98c8023
paper_authors.name: Zhoujun Li
paper_title: McEval: Massively Multilingual Code Evaluation
paper_summary: Code large language models (LLMs) have shown remarkable advances in code
  understanding, completion, and generation tasks. Programming benchmarks,
  comprised of a selection of code challenges and corresponding test cases, serve
  as a standard to evaluate the capability of different LLMs in such tasks.
  However, most existing benchmarks primarily focus on Python and are still
  restricted to a limited number of languages, where other languages are
  translated from the Python samples (e.g. MultiPL-E) degrading the data
  diversity. To further facilitate the research of code LLMs, we propose a
  massively multilingual code benchmark covering 40 programming languages
  (McEval) with 16K test samples, which substantially pushes the limits of code
  LLMs in multilingual scenarios. The benchmark contains challenging code
  completion, understanding, and generation evaluation tasks with finely curated
  massively multilingual instruction corpora McEval-Instruct. In addition, we
  introduce an effective multilingual coder mCoder trained on McEval-Instruct to
  support multilingual programming language generation. Extensive experimental
  results on McEval show that there is still a difficult journey between
  open-source models and closed-source LLMs (e.g. GPT-series models) in numerous
  languages. The instruction corpora, evaluation benchmark, and leaderboard are
  available at https://mceval.github.io/.
paper_ai_keywords: large language models (LLMs) code understanding code completion code generation tasks programming benchmarks code challenges test cases data diversity massively multilingual code benchmark McEval test samples multilingual scenarios multilingual instruction corpora McEval-Instruct multilingual coder mCoder open-source models closed-source LLMs GPT-series models