title: CodeArena: A Collective Evaluation Platform for LLM Code Generation
summary: Large Language Models (LLMs) have reshaped code generation by synergizing
  their exceptional comprehension of natural language and programming syntax,
  thereby substantially boosting developer productivity. These advancements have
  prompted numerous efforts to quantitatively evaluate their coding capabilities.
  However, persistent challenges, such as benchmark leakage, data dissipation,
  and limited system accessibility, continue to impede a timely and accurate
  assessment. To address these limitations, we introduce CodeArena, an online
  evaluation framework tailored for LLM code generation. The key innovation is a
  collective evaluation mechanism, which dynamically recalibrates individual
  model scores based on the holistic performance of all participating models,
  mitigating score biases caused by widespread benchmark leakage. In addition,
  CodeArena ensures open access to all submitted solutions and test cases and
  provides automation-friendly APIs to streamline the code evaluation workflow.
  Our main contributions are: (1) a collective evaluation system for unbiased
  assessment, (2) a public repository of solutions and test cases, and (3)
  automation-ready APIs for seamless integration.
paper_authors._id: 67c6a8b534aeb86063e94010
paper_authors.name: Mingzhe Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T08:34:49.954Z
paper_authors._id: 61711f02e0b1ddb56eb9b526
paper_authors.avatarUrl: /avatars/3e2fdf774f5bc1f73b450486d6da42d4.svg
paper_authors.fullname: Mingzhe Du
paper_authors.type: user
paper_authors.user: Elfsong
paper_authors._id: 67c6a8b534aeb86063e94011
paper_authors.name: Anh Tuan Luu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T11:02:20.575Z
paper_authors._id: 655722e80438e0854fae7554
paper_authors.avatarUrl: /avatars/b93a74f7c7880f9fe0f3ffb47e2aef5e.svg
paper_authors.fullname: Luu Anh Tuan
paper_authors.type: user
paper_authors.user: anhtuanluu36
paper_authors._id: 67c6a8b534aeb86063e94012
paper_authors.name: Bin Ji
paper_authors._id: 67c6a8b534aeb86063e94013
paper_authors.name: Xiaobao Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T11:02:48.996Z
paper_authors._id: 64cb02869e30a46f7b80b355
paper_authors.avatarUrl: /avatars/81ce4ba78826b54f0e1b53eeaff87ee6.svg
paper_authors.fullname: Xiaobao Wu
paper_authors.type: user
paper_authors.user: bobxwu
paper_authors._id: 67c6a8b534aeb86063e94014
paper_authors.name: Dong Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T11:02:43.013Z
paper_authors._id: 67c56a7f083bb2c50254bbe5
paper_authors.avatarUrl: /avatars/bdf6fd8934c2199ff169b178f6482773.svg
paper_authors.fullname: Huang, Dong
paper_authors.type: user
paper_authors.user: DongHuang-ebay
paper_authors._id: 67c6a8b534aeb86063e94015
paper_authors.name: Terry Yue Zhuo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T11:02:33.977Z
paper_authors._id: 62b7fb545233925f253531c8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b7fb545233925f253531c8/W50u2G1HK3EtUKHRU189V.jpeg
paper_authors.fullname: Terry Yue Zhuo
paper_authors.type: user
paper_authors.user: terryyz
paper_authors._id: 67c6a8b534aeb86063e94016
paper_authors.name: Qian Liu
paper_authors._id: 67c6a8b534aeb86063e94017
paper_authors.name: See-Kiong Ng
paper_title: CodeArena: A Collective Evaluation Platform for LLM Code Generation
paper_summary: Large Language Models (LLMs) have reshaped code generation by synergizing
  their exceptional comprehension of natural language and programming syntax,
  thereby substantially boosting developer productivity. These advancements have
  prompted numerous efforts to quantitatively evaluate their coding capabilities.
  However, persistent challenges, such as benchmark leakage, data dissipation,
  and limited system accessibility, continue to impede a timely and accurate
  assessment. To address these limitations, we introduce CodeArena, an online
  evaluation framework tailored for LLM code generation. The key innovation is a
  collective evaluation mechanism, which dynamically recalibrates individual
  model scores based on the holistic performance of all participating models,
  mitigating score biases caused by widespread benchmark leakage. In addition,
  CodeArena ensures open access to all submitted solutions and test cases and
  provides automation-friendly APIs to streamline the code evaluation workflow.
  Our main contributions are: (1) a collective evaluation system for unbiased
  assessment, (2) a public repository of solutions and test cases, and (3)
  automation-ready APIs for seamless integration.
paper_ai_keywords: Large Language Models (LLMs) code generation natural language comprehension programming syntax benchmark leakage collective evaluation mechanism holistic performance score biases automation-friendly APIs