title: LiveBench: A Challenging, Contamination-Free LLM Benchmark
summary: Test set contamination, wherein test data from a benchmark ends up in a newer
  model's training set, is a well-documented obstacle for fair LLM evaluation and
  can quickly render benchmarks obsolete. To mitigate this, many recent
  benchmarks crowdsource new prompts and evaluations from human or LLM judges;
  however, these can introduce significant biases, and break down when scoring
  hard questions. In this work, we introduce a new benchmark for LLMs designed to
  be immune to both test set contamination and the pitfalls of LLM judging and
  human crowdsourcing. We release LiveBench, the first benchmark that (1)
  contains frequently-updated questions from recent information sources, (2)
  scores answers automatically according to objective ground-truth values, and
  (3) contains a wide variety of challenging tasks, spanning math, coding,
  reasoning, language, instruction following, and data analysis. To achieve this,
  LiveBench contains questions that are based on recently-released math
  competitions, arXiv papers, news articles, and datasets, and it contains
  harder, contamination-free versions of tasks from previous benchmarks such as
  Big-Bench Hard, AMPS, and IFEval. We evaluate many prominent closed-source
  models, as well as dozens of open-source models ranging from 0.5B to 110B in
  size. LiveBench is difficult, with top models achieving below 65% accuracy. We
  release all questions, code, and model answers. Questions will be added and
  updated on a monthly basis, and we will release new tasks and harder versions
  of tasks over time so that LiveBench can distinguish between the capabilities
  of LLMs as they improve in the future. We welcome community engagement and
  collaboration for expanding the benchmark tasks and models.
paper_authors._id: 667e723a685216449bd881ad
paper_authors.name: Colin White
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-28T16:47:45.586Z
paper_authors._id: 65c18d3132ff878c505dc067
paper_authors.avatarUrl: /avatars/f5822f3bb4691b3635e2dc4e939da89d.svg
paper_authors.fullname: Colin White
paper_authors.type: user
paper_authors.user: colin-abacus
paper_authors._id: 667e723a685216449bd881ae
paper_authors.name: Samuel Dooley
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-28T14:58:13.864Z
paper_authors._id: 6387c1e8a79d1656d2efeebf
paper_authors.avatarUrl: /avatars/4225df1688219945b19d262d5b44377d.svg
paper_authors.fullname: Samuel
paper_authors.type: user
paper_authors.user: spamueldooley
paper_authors._id: 667e723a685216449bd881af
paper_authors.name: Manley Roberts
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:09:05.398Z
paper_authors._id: 6532d87c3e385cfc600d0416
paper_authors.avatarUrl: /avatars/c5687ab72bb8c3e865573a3047d48800.svg
paper_authors.fullname: Manley Roberts
paper_authors.type: user
paper_authors.user: manleyroberts
paper_authors._id: 667e723a685216449bd881b0
paper_authors.name: Arka Pal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:09:15.951Z
paper_authors._id: 64c14f6b02e1f8f67c73bd05
paper_authors.avatarUrl: /avatars/55d71dc415b8c1f6029ecb233abb866f.svg
paper_authors.fullname: Arka Pal
paper_authors.type: user
paper_authors.user: ArkaAbacus
paper_authors._id: 667e723a685216449bd881b1
paper_authors.name: Ben Feuer
paper_authors._id: 667e723a685216449bd881b2
paper_authors.name: Siddhartha Jain
paper_authors._id: 667e723a685216449bd881b3
paper_authors.name: Ravid Shwartz-Ziv
paper_authors._id: 667e723a685216449bd881b4
paper_authors.name: Neel Jain
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:09:40.850Z
paper_authors._id: 63e2b1ec282ee5f9624cfbcb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e2b1ec282ee5f9624cfbcb/4SVTp93cvRevacoJgiXzS.jpeg
paper_authors.fullname: Neel Jain
paper_authors.type: user
paper_authors.user: nsjain
paper_authors._id: 667e723a685216449bd881b5
paper_authors.name: Khalid Saifullah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:11:55.071Z
paper_authors._id: 603406de3d343696d96ef2a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/603406de3d343696d96ef2a6/FS3xGIC9G_JpyQ806VAyc.jpeg
paper_authors.fullname: Khalid Saifullah
paper_authors.type: user
paper_authors.user: khalidsaifullaah
paper_authors._id: 667e723a685216449bd881b6
paper_authors.name: Siddartha Naidu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:11:09.508Z
paper_authors._id: 64c14f95cac5f9ba52bbcd7f
paper_authors.avatarUrl: /avatars/98a21dc6874414305e6479d6d2ea5bef.svg
paper_authors.fullname: Siddartha Naidu
paper_authors.type: user
paper_authors.user: siddartha-abacus
paper_authors._id: 667e723a685216449bd881b7
paper_authors.name: Chinmay Hegde
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:11:02.411Z
paper_authors._id: 631620f6894404e25068856f
paper_authors.avatarUrl: /avatars/52c30caa0ee11347f82420a14ec19996.svg
paper_authors.fullname: Chinmay Hegde
paper_authors.type: user
paper_authors.user: chegde
paper_authors._id: 667e723a685216449bd881b8
paper_authors.name: Yann LeCun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:08:51.180Z
paper_authors._id: 64ed0b8c2203a126eb1a5b9a
paper_authors.avatarUrl: /avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
paper_authors.fullname: Yann LeCun
paper_authors.type: user
paper_authors.user: ylecun
paper_authors._id: 667e723a685216449bd881b9
paper_authors.name: Tom Goldstein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:10:38.927Z
paper_authors._id: 6381ca7d65dc156aba0b933d
paper_authors.avatarUrl: /avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
paper_authors.fullname: Tom Goldstein
paper_authors.type: user
paper_authors.user: tomgoldstein
paper_authors._id: 667e723a685216449bd881ba
paper_authors.name: Willie Neiswanger
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:10:31.811Z
paper_authors._id: 644bf65522d211df6444a7f4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644bf65522d211df6444a7f4/k_ddZdQDg2fzhwjI1EXyx.jpeg
paper_authors.fullname: Willie Neiswanger
paper_authors.type: user
paper_authors.user: willieneis
paper_authors._id: 667e723a685216449bd881bb
paper_authors.name: Micah Goldblum
paper_title: LiveBench: A Challenging, Contamination-Free LLM Benchmark
paper_summary: Test set contamination, wherein test data from a benchmark ends up in a newer
  model's training set, is a well-documented obstacle for fair LLM evaluation and
  can quickly render benchmarks obsolete. To mitigate this, many recent
  benchmarks crowdsource new prompts and evaluations from human or LLM judges;
  however, these can introduce significant biases, and break down when scoring
  hard questions. In this work, we introduce a new benchmark for LLMs designed to
  be immune to both test set contamination and the pitfalls of LLM judging and
  human crowdsourcing. We release LiveBench, the first benchmark that (1)
  contains frequently-updated questions from recent information sources, (2)
  scores answers automatically according to objective ground-truth values, and
  (3) contains a wide variety of challenging tasks, spanning math, coding,
  reasoning, language, instruction following, and data analysis. To achieve this,
  LiveBench contains questions that are based on recently-released math
  competitions, arXiv papers, news articles, and datasets, and it contains
  harder, contamination-free versions of tasks from previous benchmarks such as
  Big-Bench Hard, AMPS, and IFEval. We evaluate many prominent closed-source
  models, as well as dozens of open-source models ranging from 0.5B to 110B in
  size. LiveBench is difficult, with top models achieving below 65% accuracy. We
  release all questions, code, and model answers. Questions will be added and
  updated on a monthly basis, and we will release new tasks and harder versions
  of tasks over time so that LiveBench can distinguish between the capabilities
  of LLMs as they improve in the future. We welcome community engagement and
  collaboration for expanding the benchmark tasks and models.
paper_ai_keywords: test set contamination LLM evaluation benchmarks human crowdsourcing AI-generated image generation LLM judging LiveBench frequently-updated questions objective ground-truth values math coding reasoning language instruction following data analysis Big-Bench Hard AMPS IFEval closed-source models open-source models