CodeElo: Benchmarking Competition-level Code Generation of LLMs with
  Human-comparable Elo Ratings
With the increasing code reasoning capabilities of existing large language
models (LLMs) and breakthroughs in reasoning models like OpenAI o1 and o3,
there is a growing need to develop more challenging and comprehensive
benchmarks that effectively test their sophisticated competition-level coding
abilities. Existing benchmarks, like LiveCodeBench and USACO, fall short due to
the unavailability of private test cases, lack of support for special judges,
and misaligned execution environments. To bridge this gap, we introduce
CodeElo, a standardized competition-level code generation benchmark that
effectively addresses all these challenges for the first time. CodeElo
benchmark is mainly based on the official CodeForces platform and tries to
align with the platform as much as possible. We compile the recent six months
of contest problems on CodeForces with detailed information such as contest
divisions, problem difficulty ratings, and problem algorithm tags. We introduce
a unique judging method in which problems are submitted directly to the
platform and develop a reliable Elo rating calculation system that aligns with
the platform and is comparable with human participants but has lower variance.
By testing on our CodeElo, we provide the Elo ratings of 30 existing popular
open-source and 3 proprietary LLMs for the first time. The results show that
o1-mini and QwQ-32B-Preview stand out significantly, achieving Elo ratings of
1578 and 1261, respectively, while other models struggle even with the easiest
problems, placing in the lowest 20 percent among all human participants.
Detailed analysis experiments are also conducted to provide insights into
performance across algorithms and comparisons between using C++ and Python,
which can suggest directions for future studies.
6777515fab01c44c478e51e9
Shanghaoran Quan
claimed_verified
2025-01-03T14:03:42.563Z
64b9954845ce8d7ad607c14d
https://cdn-avatars.huggingface.co/v1/production/uploads/64b9954845ce8d7ad607c14d/LZ9yeTOz4J_YKrnGkcmnL.jpeg
Shanghaoran Quan
user
quanshr
6777515fab01c44c478e51ea
Jiaxi Yang
admin_assigned
2025-01-06T14:00:49.078Z
646df403ad20c6fa4f30b7ec
https://cdn-avatars.huggingface.co/v1/production/uploads/646df403ad20c6fa4f30b7ec/Q64-XMghOcBoo3itZDGYA.jpeg
Jiaxi Yang
user
jx-yang
6777515fab01c44c478e51eb
Bowen Yu
admin_assigned
2025-01-06T14:00:25.120Z
6438b43ab2ea24b52ebac2b9
/avatars/84133cd719a4b1e2f5c1a74178425f86.svg
Bowen Yu
user
bwy
6777515fab01c44c478e51ec
Bo Zheng
admin_assigned
2025-01-06T14:00:09.591Z
62c695ad5aae1c624ca992e2
/avatars/20d10fb3338e4bd4dc59e88a18cb2617.svg
Bo Zheng
user
bzheng
6777515fab01c44c478e51ed
Dayiheng Liu
admin_assigned
2025-01-06T14:00:03.244Z
6434d4989bd5a84b5dd0b0f5
https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
Dayiheng Liu
user
Losin94
6777515fab01c44c478e51ee
An Yang
claimed_verified
2025-05-19T06:43:10.402Z
62088594a5943c8a8fc94560
https://cdn-avatars.huggingface.co/v1/production/uploads/1644733028938-62088594a5943c8a8fc94560.png
An Yang
user
yangapku
6777515fab01c44c478e51ef
Xuancheng Ren
6777515fab01c44c478e51f0
Bofei Gao
admin_assigned
2025-01-06T13:59:37.033Z
65ae21adabf6d1ccb795e9a4
/avatars/b5dced62c6a3564095a8fa0959bc06cb.svg
Bofei Gao
user
KbsdJames
6777515fab01c44c478e51f1
Yibo Miao
admin_assigned
2025-01-06T13:59:30.930Z
64a139c098fad0c8a5a627a4
https://cdn-avatars.huggingface.co/v1/production/uploads/64a139c098fad0c8a5a627a4/MgWortQS64cJOZ3pL4WyH.jpeg
Yibo Miao
user
instro
6777515fab01c44c478e51f2
Yunlong Feng
claimed_verified
2025-01-03T14:03:38.392Z
5df83428da6d0311fd3d5404
https://cdn-avatars.huggingface.co/v1/production/uploads/1613644244177-5df83428da6d0311fd3d5404.jpeg
Feng YunLong
user
ylfeng
6777515fab01c44c478e51f3
Zekun Wang
admin_assigned
2025-01-06T13:59:23.832Z
656832dfbd65fd41ee7aa8cd
https://cdn-avatars.huggingface.co/v1/production/uploads/656832dfbd65fd41ee7aa8cd/HHkyetTqNq1wIBPipzjQA.jpeg
Zekun Wang
user
kugwzk
6777515fab01c44c478e51f4
Jian Yang
6777515fab01c44c478e51f5
Zeyu Cui
admin_assigned
2025-01-06T13:59:06.605Z
672c25ca8cfb61188128eb6f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/FJWy9Tt7UQmu9KcTOx3Rt.png
Zeyu Cui
user
misakamage
6777515fab01c44c478e51f6
Yang Fan
6777515fab01c44c478e51f7
Yichang Zhang
6777515fab01c44c478e51f8
Binyuan Hui
claimed_verified
2025-01-03T14:03:40.621Z
61e4c4ca1ab24785ac11ba69
https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
Binyuan Hui
user
huybery
6777515fab01c44c478e51f9
Junyang Lin
admin_assigned
2025-01-06T13:58:48.599Z
620760a26e3b7210c2ff1943
https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
Junyang Lin
user
JustinLin610
CodeElo: Benchmarking Competition-level Code Generation of LLMs with
  Human-comparable Elo Ratings
With the increasing code reasoning capabilities of existing large language
models (LLMs) and breakthroughs in reasoning models like OpenAI o1 and o3,
there is a growing need to develop more challenging and comprehensive
benchmarks that effectively test their sophisticated competition-level coding
abilities. Existing benchmarks, like LiveCodeBench and USACO, fall short due to
the unavailability of private test cases, lack of support for special judges,
and misaligned execution environments. To bridge this gap, we introduce
CodeElo, a standardized competition-level code generation benchmark that
effectively addresses all these challenges for the first time. CodeElo
benchmark is mainly based on the official CodeForces platform and tries to
align with the platform as much as possible. We compile the recent six months
of contest problems on CodeForces with detailed information such as contest
divisions, problem difficulty ratings, and problem algorithm tags. We introduce
a unique judging method in which problems are submitted directly to the
platform and develop a reliable Elo rating calculation system that aligns with
the platform and is comparable with human participants but has lower variance.
By testing on our CodeElo, we provide the Elo ratings of 30 existing popular
open-source and 3 proprietary LLMs for the first time. The results show that
o1-mini and QwQ-32B-Preview stand out significantly, achieving Elo ratings of
1578 and 1261, respectively, while other models struggle even with the easiest
problems, placing in the lowest 20 percent among all human participants.
Detailed analysis experiments are also conducted to provide insights into
performance across algorithms and comparisons between using C++ and Python,
which can suggest directions for future studies.
CodeElo is a standardized code generation benchmark based on CodeForces contests, providing comprehensive evaluation of LLMs' coding abilities with reliable Elo ratings.
large language models LLMs OpenAI o1 OpenAI o3 LiveCodeBench USACO CodeElo CodeForces Elo rating algorithm tags competition-level code generation