LiveBench: A Challenging, Contamination-Free LLM Benchmark
Test set contamination, wherein test data from a benchmark ends up in a newer
model's training set, is a well-documented obstacle for fair LLM evaluation and
can quickly render benchmarks obsolete. To mitigate this, many recent
benchmarks crowdsource new prompts and evaluations from human or LLM judges;
however, these can introduce significant biases, and break down when scoring
hard questions. In this work, we introduce a new benchmark for LLMs designed to
be immune to both test set contamination and the pitfalls of LLM judging and
human crowdsourcing. We release LiveBench, the first benchmark that (1)
contains frequently-updated questions from recent information sources, (2)
scores answers automatically according to objective ground-truth values, and
(3) contains a wide variety of challenging tasks, spanning math, coding,
reasoning, language, instruction following, and data analysis. To achieve this,
LiveBench contains questions that are based on recently-released math
competitions, arXiv papers, news articles, and datasets, and it contains
harder, contamination-free versions of tasks from previous benchmarks such as
Big-Bench Hard, AMPS, and IFEval. We evaluate many prominent closed-source
models, as well as dozens of open-source models ranging from 0.5B to 110B in
size. LiveBench is difficult, with top models achieving below 65% accuracy. We
release all questions, code, and model answers. Questions will be added and
updated on a monthly basis, and we will release new tasks and harder versions
of tasks over time so that LiveBench can distinguish between the capabilities
of LLMs as they improve in the future. We welcome community engagement and
collaboration for expanding the benchmark tasks and models.
667e723a685216449bd881ad
Colin White
extracted_confirmed
2024-06-28T16:47:45.586Z
65c18d3132ff878c505dc067
/avatars/f5822f3bb4691b3635e2dc4e939da89d.svg
Colin White
user
colin-abacus
667e723a685216449bd881ae
Samuel Dooley
claimed_verified
2024-06-28T14:58:13.864Z
6387c1e8a79d1656d2efeebf
/avatars/4225df1688219945b19d262d5b44377d.svg
Samuel
user
spamueldooley
667e723a685216449bd881af
Manley Roberts
admin_assigned
2024-06-28T09:09:05.398Z
6532d87c3e385cfc600d0416
/avatars/c5687ab72bb8c3e865573a3047d48800.svg
Manley Roberts
user
manleyroberts
667e723a685216449bd881b0
Arka Pal
admin_assigned
2024-06-28T09:09:15.951Z
64c14f6b02e1f8f67c73bd05
/avatars/55d71dc415b8c1f6029ecb233abb866f.svg
Arka Pal
user
ArkaAbacus
667e723a685216449bd881b1
Ben Feuer
claimed_verified
2025-06-07T05:51:15.245Z
62f7f4efe7c1c9bf10c81465
https://cdn-avatars.huggingface.co/v1/production/uploads/62f7f4efe7c1c9bf10c81465/AYlOg0fkP1o4GAP-8Y3xt.jpeg
Benjamin Feuer
user
penfever
667e723a685216449bd881b2
Siddhartha Jain
667e723a685216449bd881b3
Ravid Shwartz-Ziv
667e723a685216449bd881b4
Neel Jain
admin_assigned
2024-06-28T09:09:40.850Z
63e2b1ec282ee5f9624cfbcb
https://cdn-avatars.huggingface.co/v1/production/uploads/63e2b1ec282ee5f9624cfbcb/4SVTp93cvRevacoJgiXzS.jpeg
Neel Jain
user
nsjain
667e723a685216449bd881b5
Khalid Saifullah
admin_assigned
2024-06-28T09:11:55.071Z
603406de3d343696d96ef2a6
https://cdn-avatars.huggingface.co/v1/production/uploads/603406de3d343696d96ef2a6/FS3xGIC9G_JpyQ806VAyc.jpeg
Khalid Saifullah
user
khalidsaifullaah
667e723a685216449bd881b6
Siddartha Naidu
admin_assigned
2024-06-28T09:11:09.508Z
64c14f95cac5f9ba52bbcd7f
/avatars/98a21dc6874414305e6479d6d2ea5bef.svg
Siddartha Naidu
user
siddartha-abacus
667e723a685216449bd881b7
Chinmay Hegde
admin_assigned
2024-06-28T09:11:02.411Z
631620f6894404e25068856f
/avatars/52c30caa0ee11347f82420a14ec19996.svg
Chinmay Hegde
user
chegde
667e723a685216449bd881b8
Yann LeCun
admin_assigned
2024-06-28T09:08:51.180Z
64ed0b8c2203a126eb1a5b9a
/avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
Yann LeCun
user
ylecun
667e723a685216449bd881b9
Tom Goldstein
admin_assigned
2024-06-28T09:10:38.927Z
6381ca7d65dc156aba0b933d
/avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
Tom Goldstein
user
tomgoldstein
667e723a685216449bd881ba
Willie Neiswanger
admin_assigned
2024-06-28T09:10:31.811Z
644bf65522d211df6444a7f4
https://cdn-avatars.huggingface.co/v1/production/uploads/644bf65522d211df6444a7f4/k_ddZdQDg2fzhwjI1EXyx.jpeg
Willie Neiswanger
user
willieneis
667e723a685216449bd881bb
Micah Goldblum
LiveBench: A Challenging, Contamination-Free LLM Benchmark
Test set contamination, wherein test data from a benchmark ends up in a newer
model's training set, is a well-documented obstacle for fair LLM evaluation and
can quickly render benchmarks obsolete. To mitigate this, many recent
benchmarks crowdsource new prompts and evaluations from human or LLM judges;
however, these can introduce significant biases, and break down when scoring
hard questions. In this work, we introduce a new benchmark for LLMs designed to
be immune to both test set contamination and the pitfalls of LLM judging and
human crowdsourcing. We release LiveBench, the first benchmark that (1)
contains frequently-updated questions from recent information sources, (2)
scores answers automatically according to objective ground-truth values, and
(3) contains a wide variety of challenging tasks, spanning math, coding,
reasoning, language, instruction following, and data analysis. To achieve this,
LiveBench contains questions that are based on recently-released math
competitions, arXiv papers, news articles, and datasets, and it contains
harder, contamination-free versions of tasks from previous benchmarks such as
Big-Bench Hard, AMPS, and IFEval. We evaluate many prominent closed-source
models, as well as dozens of open-source models ranging from 0.5B to 110B in
size. LiveBench is difficult, with top models achieving below 65% accuracy. We
release all questions, code, and model answers. Questions will be added and
updated on a monthly basis, and we will release new tasks and harder versions
of tasks over time so that LiveBench can distinguish between the capabilities
of LLMs as they improve in the future. We welcome community engagement and
collaboration for expanding the benchmark tasks and models.
LiveBench is a benchmark designed to be contamination-free and bias-resilient, featuring automatically scored questions from various sources and encompassing diverse tasks including math, coding, reasoning, and data analysis.
test set contamination LLMs benchmarks LiveBench Big-Bench Hard AMPS IFEval closed-source models open-source models