The Leaderboard Illusion
Measuring progress is fundamental to the advancement of any scientific field.
As benchmarks play an increasingly central role, they also grow more
susceptible to distortion. Chatbot Arena has emerged as the go-to leaderboard
for ranking the most capable AI systems. Yet, in this work we identify
systematic issues that have resulted in a distorted playing field. We find that
undisclosed private testing practices benefit a handful of providers who are
able to test multiple variants before public release and retract scores if
desired. We establish that the ability of these providers to choose the best
score leads to biased Arena scores due to selective disclosure of performance
results. At an extreme, we identify 27 private LLM variants tested by Meta in
the lead-up to the Llama-4 release. We also establish that proprietary closed
models are sampled at higher rates (number of battles) and have fewer models
removed from the arena than open-weight and open-source alternatives. Both
these policies lead to large data access asymmetries over time. Providers like
Google and OpenAI have received an estimated 19.2% and 20.4% of all data on the
arena, respectively. In contrast, a combined 83 open-weight models have only
received an estimated 29.7% of the total data. We show that access to Chatbot
Arena data yields substantial benefits; even limited additional data can result
in relative performance gains of up to 112% on the arena distribution, based on
our conservative estimates. Together, these dynamics result in overfitting to
Arena-specific dynamics rather than general model quality. The Arena builds on
the substantial efforts of both the organizers and an open community that
maintains this valuable evaluation platform. We offer actionable
recommendations to reform the Chatbot Arena's evaluation framework and promote
fairer, more transparent benchmarking for the field
6811ae6b7f4f553788e905b8
Shivalika Singh
admin_assigned
2025-04-30T12:19:56.166Z
62c2175d756039dd0dd20509
https://cdn-avatars.huggingface.co/v1/production/uploads/1677753847837-62c2175d756039dd0dd20509.jpeg
Shivalika Singh
user
shivalikasingh
6811ae6b7f4f553788e905b9
Yiyang Nan
admin_assigned
2025-04-30T12:20:27.669Z
66ad9237091a147c4f154dbb
/avatars/c80258e877bafc08452f241a6ecfec04.svg
Yiyang Nan
user
olivernan
6811ae6b7f4f553788e905ba
Alex Wang
admin_assigned
2025-04-30T12:20:38.051Z
62fdf36a594a7b92e671e3b5
/avatars/d3e0b109f4fc7da45873d599659ae5b5.svg
Wang
user
AlexWang
6811ae6b7f4f553788e905bb
Daniel D'Souza
admin_assigned
2025-04-30T12:20:45.388Z
6658011eaba105a066e37e1b
https://cdn-avatars.huggingface.co/v1/production/uploads/6658011eaba105a066e37e1b/VPwyTv1bnVMQbVMoMQzcf.jpeg
Daniel D'souza
user
dsouzadaniel
6811ae6b7f4f553788e905bc
Sayash Kapoor
admin_assigned
2025-04-30T12:20:51.466Z
630906fde3246e2be88bfab3
/avatars/bca882c027d17f7feba837baae71ec2d.svg
Sayash Kapoor
user
sayashk
6811ae6b7f4f553788e905bd
Ahmet Üstün
admin_assigned
2025-04-30T12:20:58.423Z
60d35d7ad7b174177faabd5b
/avatars/4e5403b9d4a845a2d21e8217dc3c16d2.svg
Ahmet Üstün
user
ahmetu
6811ae6b7f4f553788e905be
Sanmi Koyejo
admin_assigned
2025-04-30T12:21:04.000Z
64931e7e2da595588288f161
https://cdn-avatars.huggingface.co/v1/production/uploads/64931e7e2da595588288f161/4jOhJOFsU7RVFMgGk5kO7.jpeg
Sanmi Koyejo
user
sanmikoyejo
6811ae6b7f4f553788e905bf
Yuntian Deng
claimed_verified
2025-04-30T09:56:42.033Z
63081e15a670ed10f9d44229
https://cdn-avatars.huggingface.co/v1/production/uploads/63081e15a670ed10f9d44229/w1b9uq-9774bMMgJbSPsS.jpeg
Yuntian Deng
user
yuntian-deng
6811ae6b7f4f553788e905c0
Shayne Longpre
admin_assigned
2025-04-30T12:21:11.011Z
61f4283a81c4d30f58140242
/avatars/a1cf1ef1fd442c36ed65c68e51919fed.svg
Shayne Longpre
user
Shayne
6811ae6b7f4f553788e905c1
Noah Smith
6811ae6b7f4f553788e905c2
Beyza Ermis
admin_assigned
2025-04-30T12:21:23.787Z
6634e8c750ea73bfbc59223e
/avatars/9f0a7bcdfd82d5344932437d458fce9f.svg
Beyza Ermis
user
beyzaermis
6811ae6b7f4f553788e905c3
Marzieh Fadaee
admin_assigned
2025-04-30T12:21:31.242Z
6441042d5d600fb0951a5f99
https://cdn-avatars.huggingface.co/v1/production/uploads/6441042d5d600fb0951a5f99/4CbOaYcEz99BtVAQvnGTn.jpeg
Marzieh Fadaee
user
MarziehFadaee
6811ae6b7f4f553788e905c4
Sara Hooker
admin_assigned
2025-04-30T12:21:37.932Z
63434eb76f59b79da07dbddf
https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
Sara Hooker
user
sarahooker
The Leaderboard Illusion
Measuring progress is fundamental to the advancement of any scientific field.
As benchmarks play an increasingly central role, they also grow more
susceptible to distortion. Chatbot Arena has emerged as the go-to leaderboard
for ranking the most capable AI systems. Yet, in this work we identify
systematic issues that have resulted in a distorted playing field. We find that
undisclosed private testing practices benefit a handful of providers who are
able to test multiple variants before public release and retract scores if
desired. We establish that the ability of these providers to choose the best
score leads to biased Arena scores due to selective disclosure of performance
results. At an extreme, we identify 27 private LLM variants tested by Meta in
the lead-up to the Llama-4 release. We also establish that proprietary closed
models are sampled at higher rates (number of battles) and have fewer models
removed from the arena than open-weight and open-source alternatives. Both
these policies lead to large data access asymmetries over time. Providers like
Google and OpenAI have received an estimated 19.2% and 20.4% of all data on the
arena, respectively. In contrast, a combined 83 open-weight models have only
received an estimated 29.7% of the total data. We show that access to Chatbot
Arena data yields substantial benefits; even limited additional data can result
in relative performance gains of up to 112% on the arena distribution, based on
our conservative estimates. Together, these dynamics result in overfitting to
Arena-specific dynamics rather than general model quality. The Arena builds on
the substantial efforts of both the organizers and an open community that
maintains this valuable evaluation platform. We offer actionable
recommendations to reform the Chatbot Arena's evaluation framework and promote
fairer, more transparent benchmarking for the field