title: Position: AI Competitions Provide the Gold Standard for Empirical Rigor
    in GenAI Evaluation
summary: In this position paper, we observe that empirical evaluation in Generative AI
  is at a crisis point since traditional ML evaluation and benchmarking
  strategies are insufficient to meet the needs of evaluating modern GenAI models
  and systems. There are many reasons for this, including the fact that these
  models typically have nearly unbounded input and output spaces, typically do
  not have a well defined ground truth target, and typically exhibit strong
  feedback loops and prediction dependence based on context of previous model
  outputs. On top of these critical issues, we argue that the problems of {\em
  leakage} and {\em contamination} are in fact the most important and difficult
  issues to address for GenAI evaluations. Interestingly, the field of AI
  Competitions has developed effective measures and practices to combat leakage
  for the purpose of counteracting cheating by bad actors within a competition
  setting. This makes AI Competitions an especially valuable (but underutilized)
  resource. Now is time for the field to view AI Competitions as the gold
  standard for empirical rigor in GenAI evaluation, and to harness and harvest
  their results with according value.
paper_authors._id: 68233e50c28353126f22bf74
paper_authors.name: D. Sculley
paper_authors._id: 68233e50c28353126f22bf75
paper_authors.name: Will Cukierski
paper_authors._id: 68233e50c28353126f22bf76
paper_authors.name: Phil Culliton
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-14T07:35:34.570Z
paper_authors._id: 627a9aa0f967ef475abe8037
paper_authors.avatarUrl: /avatars/47cf94d6c015577d0a860b617c9ea2b8.svg
paper_authors.fullname: Phil Culliton
paper_authors.type: user
paper_authors.user: PhilCulliton
paper_authors._id: 68233e50c28353126f22bf77
paper_authors.name: Sohier Dane
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-14T07:35:40.368Z
paper_authors._id: 6823721744c921f21ffc23dd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/BZQBui3e4JMVGUzBgQBC3.png
paper_authors.fullname: Sohier Dane
paper_authors.type: user
paper_authors.user: SohierDane
paper_authors._id: 68233e50c28353126f22bf78
paper_authors.name: Maggie Demkin
paper_authors._id: 68233e50c28353126f22bf79
paper_authors.name: Ryan Holbrook
paper_authors._id: 68233e50c28353126f22bf7a
paper_authors.name: Addison Howard
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-14T07:35:49.163Z
paper_authors._id: 68234703b30c1a41e2ea1387
paper_authors.avatarUrl: /avatars/5842f616f01f970696227d6eeb83a30b.svg
paper_authors.fullname: Addison Howard
paper_authors.type: user
paper_authors.user: ahoward14
paper_authors._id: 68233e50c28353126f22bf7b
paper_authors.name: Paul Mooney
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-14T07:35:37.635Z
paper_authors._id: 627a89f393d0b645835e57aa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652197873133-noauth.jpeg
paper_authors.fullname: Paul M
paper_authors.type: user
paper_authors.user: paultimothymooney
paper_authors._id: 68233e50c28353126f22bf7c
paper_authors.name: Walter Reade
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-14T07:35:46.514Z
paper_authors._id: 63bf3b2287619d1458cddbe0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1673476847649-noauth.jpeg
paper_authors.fullname: Walter Reade
paper_authors.type: user
paper_authors.user: walterreade
paper_authors._id: 68233e50c28353126f22bf7d
paper_authors.name: Megan Risdal
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-13T13:17:20.366Z
paper_authors._id: 626b23fc822e3d85324babed
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626b23fc822e3d85324babed/CbRuisnyy5JsCeA5uxYL7.jpeg
paper_authors.fullname: Meg Risdal
paper_authors.type: user
paper_authors.user: megrisdal
paper_authors._id: 68233e50c28353126f22bf7e
paper_authors.name: Nate Keating
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-14T07:35:43.363Z
paper_authors._id: 621656e913bc8271a06798a0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/621656e913bc8271a06798a0/2cmQEKQG3fuhKy45WLPoQ.jpeg
paper_authors.fullname: Nate Keating
paper_authors.type: user
paper_authors.user: natekeating
paper_title: Position: AI Competitions Provide the Gold Standard for Empirical Rigor
    in GenAI Evaluation
paper_summary: In this position paper, we observe that empirical evaluation in Generative AI
  is at a crisis point since traditional ML evaluation and benchmarking
  strategies are insufficient to meet the needs of evaluating modern GenAI models
  and systems. There are many reasons for this, including the fact that these
  models typically have nearly unbounded input and output spaces, typically do
  not have a well defined ground truth target, and typically exhibit strong
  feedback loops and prediction dependence based on context of previous model
  outputs. On top of these critical issues, we argue that the problems of {\em
  leakage} and {\em contamination} are in fact the most important and difficult
  issues to address for GenAI evaluations. Interestingly, the field of AI
  Competitions has developed effective measures and practices to combat leakage
  for the purpose of counteracting cheating by bad actors within a competition
  setting. This makes AI Competitions an especially valuable (but underutilized)
  resource. Now is time for the field to view AI Competitions as the gold
  standard for empirical rigor in GenAI evaluation, and to harness and harvest
  their results with according value.
paper_ai_summary: Empirical evaluation of Generative AI models faces significant challenges due to unbounded input/output spaces, lack of well-defined ground truth, feedback loops, and issues like leakage and contamination, suggesting AI Competitions as a gold standard for robust evaluation.
paper_ai_keywords: Generative AI empirical evaluation ML evaluation benchmarking input/output spaces ground truth feedback loops prediction dependence leakage contamination AI Competitions