title: The Leaderboard Illusion
summary: Measuring progress is fundamental to the advancement of any scientific field.
  As benchmarks play an increasingly central role, they also grow more
  susceptible to distortion. Chatbot Arena has emerged as the go-to leaderboard
  for ranking the most capable AI systems. Yet, in this work we identify
  systematic issues that have resulted in a distorted playing field. We find that
  undisclosed private testing practices benefit a handful of providers who are
  able to test multiple variants before public release and retract scores if
  desired. We establish that the ability of these providers to choose the best
  score leads to biased Arena scores due to selective disclosure of performance
  results. At an extreme, we identify 27 private LLM variants tested by Meta in
  the lead-up to the Llama-4 release. We also establish that proprietary closed
  models are sampled at higher rates (number of battles) and have fewer models
  removed from the arena than open-weight and open-source alternatives. Both
  these policies lead to large data access asymmetries over time. Providers like
  Google and OpenAI have received an estimated 19.2% and 20.4% of all data on the
  arena, respectively. In contrast, a combined 83 open-weight models have only
  received an estimated 29.7% of the total data. We show that access to Chatbot
  Arena data yields substantial benefits; even limited additional data can result
  in relative performance gains of up to 112% on the arena distribution, based on
  our conservative estimates. Together, these dynamics result in overfitting to
  Arena-specific dynamics rather than general model quality. The Arena builds on
  the substantial efforts of both the organizers and an open community that
  maintains this valuable evaluation platform. We offer actionable
  recommendations to reform the Chatbot Arena's evaluation framework and promote
  fairer, more transparent benchmarking for the field
paper_authors._id: 6811ae6b7f4f553788e905b8
paper_authors.name: Shivalika Singh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:19:56.166Z
paper_authors._id: 62c2175d756039dd0dd20509
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677753847837-62c2175d756039dd0dd20509.jpeg
paper_authors.fullname: Shivalika Singh
paper_authors.type: user
paper_authors.user: shivalikasingh
paper_authors._id: 6811ae6b7f4f553788e905b9
paper_authors.name: Yiyang Nan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:20:27.669Z
paper_authors._id: 66ad9237091a147c4f154dbb
paper_authors.avatarUrl: /avatars/c80258e877bafc08452f241a6ecfec04.svg
paper_authors.fullname: Yiyang Nan
paper_authors.type: user
paper_authors.user: olivernan
paper_authors._id: 6811ae6b7f4f553788e905ba
paper_authors.name: Alex Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:20:38.051Z
paper_authors._id: 62fdf36a594a7b92e671e3b5
paper_authors.avatarUrl: /avatars/d3e0b109f4fc7da45873d599659ae5b5.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: AlexWang
paper_authors._id: 6811ae6b7f4f553788e905bb
paper_authors.name: Daniel D'Souza
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:20:45.388Z
paper_authors._id: 6658011eaba105a066e37e1b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6658011eaba105a066e37e1b/VPwyTv1bnVMQbVMoMQzcf.jpeg
paper_authors.fullname: Daniel D'souza
paper_authors.type: user
paper_authors.user: dsouzadaniel
paper_authors._id: 6811ae6b7f4f553788e905bc
paper_authors.name: Sayash Kapoor
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:20:51.466Z
paper_authors._id: 630906fde3246e2be88bfab3
paper_authors.avatarUrl: /avatars/bca882c027d17f7feba837baae71ec2d.svg
paper_authors.fullname: Sayash Kapoor
paper_authors.type: user
paper_authors.user: sayashk
paper_authors._id: 6811ae6b7f4f553788e905bd
paper_authors.name: Ahmet Üstün
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:20:58.423Z
paper_authors._id: 60d35d7ad7b174177faabd5b
paper_authors.avatarUrl: /avatars/4e5403b9d4a845a2d21e8217dc3c16d2.svg
paper_authors.fullname: Ahmet Üstün
paper_authors.type: user
paper_authors.user: ahmetu
paper_authors._id: 6811ae6b7f4f553788e905be
paper_authors.name: Sanmi Koyejo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:21:04.000Z
paper_authors._id: 64931e7e2da595588288f161
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64931e7e2da595588288f161/4jOhJOFsU7RVFMgGk5kO7.jpeg
paper_authors.fullname: Sanmi Koyejo
paper_authors.type: user
paper_authors.user: sanmikoyejo
paper_authors._id: 6811ae6b7f4f553788e905bf
paper_authors.name: Yuntian Deng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-30T09:56:42.033Z
paper_authors._id: 63081e15a670ed10f9d44229
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63081e15a670ed10f9d44229/w1b9uq-9774bMMgJbSPsS.jpeg
paper_authors.fullname: Yuntian Deng
paper_authors.type: user
paper_authors.user: yuntian-deng
paper_authors._id: 6811ae6b7f4f553788e905c0
paper_authors.name: Shayne Longpre
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:21:11.011Z
paper_authors._id: 61f4283a81c4d30f58140242
paper_authors.avatarUrl: /avatars/a1cf1ef1fd442c36ed65c68e51919fed.svg
paper_authors.fullname: Shayne Longpre
paper_authors.type: user
paper_authors.user: Shayne
paper_authors._id: 6811ae6b7f4f553788e905c1
paper_authors.name: Noah Smith
paper_authors._id: 6811ae6b7f4f553788e905c2
paper_authors.name: Beyza Ermis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:21:23.787Z
paper_authors._id: 6634e8c750ea73bfbc59223e
paper_authors.avatarUrl: /avatars/9f0a7bcdfd82d5344932437d458fce9f.svg
paper_authors.fullname: Beyza Ermis
paper_authors.type: user
paper_authors.user: beyzaermis
paper_authors._id: 6811ae6b7f4f553788e905c3
paper_authors.name: Marzieh Fadaee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:21:31.242Z
paper_authors._id: 6441042d5d600fb0951a5f99
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6441042d5d600fb0951a5f99/4CbOaYcEz99BtVAQvnGTn.jpeg
paper_authors.fullname: Marzieh Fadaee
paper_authors.type: user
paper_authors.user: MarziehFadaee
paper_authors._id: 6811ae6b7f4f553788e905c4
paper_authors.name: Sara Hooker
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:21:37.932Z
paper_authors._id: 63434eb76f59b79da07dbddf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63434eb76f59b79da07dbddf/BEwmVjqPNYlqmutXG0G6e.jpeg
paper_authors.fullname: Sara Hooker
paper_authors.type: user
paper_authors.user: sarahooker
paper_title: The Leaderboard Illusion
paper_summary: Measuring progress is fundamental to the advancement of any scientific field.
  As benchmarks play an increasingly central role, they also grow more
  susceptible to distortion. Chatbot Arena has emerged as the go-to leaderboard
  for ranking the most capable AI systems. Yet, in this work we identify
  systematic issues that have resulted in a distorted playing field. We find that
  undisclosed private testing practices benefit a handful of providers who are
  able to test multiple variants before public release and retract scores if
  desired. We establish that the ability of these providers to choose the best
  score leads to biased Arena scores due to selective disclosure of performance
  results. At an extreme, we identify 27 private LLM variants tested by Meta in
  the lead-up to the Llama-4 release. We also establish that proprietary closed
  models are sampled at higher rates (number of battles) and have fewer models
  removed from the arena than open-weight and open-source alternatives. Both
  these policies lead to large data access asymmetries over time. Providers like
  Google and OpenAI have received an estimated 19.2% and 20.4% of all data on the
  arena, respectively. In contrast, a combined 83 open-weight models have only
  received an estimated 29.7% of the total data. We show that access to Chatbot
  Arena data yields substantial benefits; even limited additional data can result
  in relative performance gains of up to 112% on the arena distribution, based on
  our conservative estimates. Together, these dynamics result in overfitting to
  Arena-specific dynamics rather than general model quality. The Arena builds on
  the substantial efforts of both the organizers and an open community that
  maintains this valuable evaluation platform. We offer actionable
  recommendations to reform the Chatbot Arena's evaluation framework and promote
  fairer, more transparent benchmarking for the field