title: CodeElo: Benchmarking Competition-level Code Generation of LLMs with
    Human-comparable Elo Ratings
summary: With the increasing code reasoning capabilities of existing large language
  models (LLMs) and breakthroughs in reasoning models like OpenAI o1 and o3,
  there is a growing need to develop more challenging and comprehensive
  benchmarks that effectively test their sophisticated competition-level coding
  abilities. Existing benchmarks, like LiveCodeBench and USACO, fall short due to
  the unavailability of private test cases, lack of support for special judges,
  and misaligned execution environments. To bridge this gap, we introduce
  CodeElo, a standardized competition-level code generation benchmark that
  effectively addresses all these challenges for the first time. CodeElo
  benchmark is mainly based on the official CodeForces platform and tries to
  align with the platform as much as possible. We compile the recent six months
  of contest problems on CodeForces with detailed information such as contest
  divisions, problem difficulty ratings, and problem algorithm tags. We introduce
  a unique judging method in which problems are submitted directly to the
  platform and develop a reliable Elo rating calculation system that aligns with
  the platform and is comparable with human participants but has lower variance.
  By testing on our CodeElo, we provide the Elo ratings of 30 existing popular
  open-source and 3 proprietary LLMs for the first time. The results show that
  o1-mini and QwQ-32B-Preview stand out significantly, achieving Elo ratings of
  1578 and 1261, respectively, while other models struggle even with the easiest
  problems, placing in the lowest 20 percent among all human participants.
  Detailed analysis experiments are also conducted to provide insights into
  performance across algorithms and comparisons between using C++ and Python,
  which can suggest directions for future studies.
paper_authors._id: 6777515fab01c44c478e51e9
paper_authors.name: Shanghaoran Quan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:03:42.563Z
paper_authors._id: 64b9954845ce8d7ad607c14d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b9954845ce8d7ad607c14d/LZ9yeTOz4J_YKrnGkcmnL.jpeg
paper_authors.fullname: Shanghaoran Quan
paper_authors.type: user
paper_authors.user: quanshr
paper_authors._id: 6777515fab01c44c478e51ea
paper_authors.name: Jiaxi Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T14:00:49.078Z
paper_authors._id: 646df403ad20c6fa4f30b7ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646df403ad20c6fa4f30b7ec/Q64-XMghOcBoo3itZDGYA.jpeg
paper_authors.fullname: Jiaxi Yang
paper_authors.type: user
paper_authors.user: jx-yang
paper_authors._id: 6777515fab01c44c478e51eb
paper_authors.name: Bowen Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T14:00:25.120Z
paper_authors._id: 6438b43ab2ea24b52ebac2b9
paper_authors.avatarUrl: /avatars/84133cd719a4b1e2f5c1a74178425f86.svg
paper_authors.fullname: Bowen Yu
paper_authors.type: user
paper_authors.user: bwy
paper_authors._id: 6777515fab01c44c478e51ec
paper_authors.name: Bo Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T14:00:09.591Z
paper_authors._id: 62c695ad5aae1c624ca992e2
paper_authors.avatarUrl: /avatars/20d10fb3338e4bd4dc59e88a18cb2617.svg
paper_authors.fullname: Bo Zheng
paper_authors.type: user
paper_authors.user: bzheng
paper_authors._id: 6777515fab01c44c478e51ed
paper_authors.name: Dayiheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T14:00:03.244Z
paper_authors._id: 6434d4989bd5a84b5dd0b0f5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
paper_authors.fullname: Dayiheng Liu
paper_authors.type: user
paper_authors.user: Losin94
paper_authors._id: 6777515fab01c44c478e51ee
paper_authors.name: An Yang
paper_authors._id: 6777515fab01c44c478e51ef
paper_authors.name: Xuancheng Ren
paper_authors._id: 6777515fab01c44c478e51f0
paper_authors.name: Bofei Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T13:59:37.033Z
paper_authors._id: 65ae21adabf6d1ccb795e9a4
paper_authors.avatarUrl: /avatars/b5dced62c6a3564095a8fa0959bc06cb.svg
paper_authors.fullname: Bofei Gao
paper_authors.type: user
paper_authors.user: KbsdJames
paper_authors._id: 6777515fab01c44c478e51f1
paper_authors.name: Yibo Miao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T13:59:30.930Z
paper_authors._id: 64a139c098fad0c8a5a627a4
paper_authors.avatarUrl: /avatars/6eb508abd827d4a4f5abb6b24155b22d.svg
paper_authors.fullname: Yibo Miao
paper_authors.type: user
paper_authors.user: instro
paper_authors._id: 6777515fab01c44c478e51f2
paper_authors.name: Yunlong Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:03:38.392Z
paper_authors._id: 5df83428da6d0311fd3d5404
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1613644244177-5df83428da6d0311fd3d5404.jpeg
paper_authors.fullname: Feng YunLong
paper_authors.type: user
paper_authors.user: ylfeng
paper_authors._id: 6777515fab01c44c478e51f3
paper_authors.name: Zekun Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T13:59:23.832Z
paper_authors._id: 656832dfbd65fd41ee7aa8cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/656832dfbd65fd41ee7aa8cd/HHkyetTqNq1wIBPipzjQA.jpeg
paper_authors.fullname: Zekun Wang
paper_authors.type: user
paper_authors.user: kugwzk
paper_authors._id: 6777515fab01c44c478e51f4
paper_authors.name: Jian Yang
paper_authors._id: 6777515fab01c44c478e51f5
paper_authors.name: Zeyu Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T13:59:06.605Z
paper_authors._id: 672c25ca8cfb61188128eb6f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/FJWy9Tt7UQmu9KcTOx3Rt.png
paper_authors.fullname: Zeyu Cui
paper_authors.type: user
paper_authors.user: misakamage
paper_authors._id: 6777515fab01c44c478e51f6
paper_authors.name: Yang Fan
paper_authors._id: 6777515fab01c44c478e51f7
paper_authors.name: Yichang Zhang
paper_authors._id: 6777515fab01c44c478e51f8
paper_authors.name: Binyuan Hui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:03:40.621Z
paper_authors._id: 61e4c4ca1ab24785ac11ba69
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
paper_authors.fullname: Binyuan Hui
paper_authors.type: user
paper_authors.user: huybery
paper_authors._id: 6777515fab01c44c478e51f9
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T13:58:48.599Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_title: CodeElo: Benchmarking Competition-level Code Generation of LLMs with
    Human-comparable Elo Ratings
paper_summary: With the increasing code reasoning capabilities of existing large language
  models (LLMs) and breakthroughs in reasoning models like OpenAI o1 and o3,
  there is a growing need to develop more challenging and comprehensive
  benchmarks that effectively test their sophisticated competition-level coding
  abilities. Existing benchmarks, like LiveCodeBench and USACO, fall short due to
  the unavailability of private test cases, lack of support for special judges,
  and misaligned execution environments. To bridge this gap, we introduce
  CodeElo, a standardized competition-level code generation benchmark that
  effectively addresses all these challenges for the first time. CodeElo
  benchmark is mainly based on the official CodeForces platform and tries to
  align with the platform as much as possible. We compile the recent six months
  of contest problems on CodeForces with detailed information such as contest
  divisions, problem difficulty ratings, and problem algorithm tags. We introduce
  a unique judging method in which problems are submitted directly to the
  platform and develop a reliable Elo rating calculation system that aligns with
  the platform and is comparable with human participants but has lower variance.
  By testing on our CodeElo, we provide the Elo ratings of 30 existing popular
  open-source and 3 proprietary LLMs for the first time. The results show that
  o1-mini and QwQ-32B-Preview stand out significantly, achieving Elo ratings of
  1578 and 1261, respectively, while other models struggle even with the easiest
  problems, placing in the lowest 20 percent among all human participants.
  Detailed analysis experiments are also conducted to provide insights into
  performance across algorithms and comparisons between using C++ and Python,
  which can suggest directions for future studies.