title: YourBench: Easy Custom Evaluation Sets for Everyone
summary: Evaluating large language models (LLMs) effectively remains a critical
  bottleneck, as traditional static benchmarks suffer from saturation and
  contamination, while human evaluations are costly and slow. This hinders timely
  or domain-specific assessment, crucial for real-world applications. We
  introduce YourBench, a novel, open-source framework that addresses these
  limitations by enabling dynamic, automated generation of reliable, up-to-date,
  and domain-tailored benchmarks cheaply and without manual annotation, directly
  from user-provided documents. We demonstrate its efficacy by replicating 7
  diverse MMLU subsets using minimal source text, achieving this for under 15 USD
  in total inference costs while perfectly preserving the relative model
  performance rankings (Spearman Rho = 1) observed on the original benchmark. To
  ensure that YourBench generates data grounded in provided input instead of
  relying on posterior parametric knowledge in models, we also introduce
  Tempora-0325, a novel dataset of over 7K diverse documents, published
  exclusively after March 2025. Our comprehensive analysis spans 26 SoTA models
  from 7 major families across varying scales (3-671B parameters) to validate the
  quality of generated evaluations through rigorous algorithmic checks (e.g.,
  citation grounding) and human assessments. We release the YourBench library,
  the Tempora-0325 dataset, 150k+ question answer pairs based on Tempora and all
  evaluation and inference traces to facilitate reproducible research and empower
  the community to generate bespoke benchmarks on demand, fostering more relevant
  and trustworthy LLM evaluation.
paper_authors._id: 67eddf485224e5e2c1a39491
paper_authors.name: Sumuk Shashidhar
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-31T02:59:18.819Z
paper_authors._id: 64676c81e7a6a374fd181110
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64676c81e7a6a374fd181110/HEDlAT9FYhJF9Nw__LLNI.jpeg
paper_authors.fullname: Sumuk Shashidhar
paper_authors.type: user
paper_authors.user: sumuks
paper_authors._id: 67eddf485224e5e2c1a39492
paper_authors.name: Clémentine Fourrier
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-04-03T01:07:21.792Z
paper_authors._id: 6202a599216215a22221dea9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644340617257-noauth.png
paper_authors.fullname: Clémentine Fourrier
paper_authors.type: user
paper_authors.user: clefourrier
paper_authors._id: 67eddf485224e5e2c1a39493
paper_authors.name: Alina Lozovskia
paper_authors._id: 67eddf485224e5e2c1a39494
paper_authors.name: Thomas Wolf
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T06:57:10.557Z
paper_authors._id: 5df7e9e5da6d0311fd3d53f9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1583857746553-5df7e9e5da6d0311fd3d53f9.jpeg
paper_authors.fullname: Thomas Wolf
paper_authors.type: user
paper_authors.user: thomwolf
paper_authors._id: 67eddf485224e5e2c1a39495
paper_authors.name: Gokhan Tur
paper_authors._id: 67eddf485224e5e2c1a39496
paper_authors.name: Dilek Hakkani-Tür
paper_title: YourBench: Easy Custom Evaluation Sets for Everyone
paper_summary: Evaluating large language models (LLMs) effectively remains a critical
  bottleneck, as traditional static benchmarks suffer from saturation and
  contamination, while human evaluations are costly and slow. This hinders timely
  or domain-specific assessment, crucial for real-world applications. We
  introduce YourBench, a novel, open-source framework that addresses these
  limitations by enabling dynamic, automated generation of reliable, up-to-date,
  and domain-tailored benchmarks cheaply and without manual annotation, directly
  from user-provided documents. We demonstrate its efficacy by replicating 7
  diverse MMLU subsets using minimal source text, achieving this for under 15 USD
  in total inference costs while perfectly preserving the relative model
  performance rankings (Spearman Rho = 1) observed on the original benchmark. To
  ensure that YourBench generates data grounded in provided input instead of
  relying on posterior parametric knowledge in models, we also introduce
  Tempora-0325, a novel dataset of over 7K diverse documents, published
  exclusively after March 2025. Our comprehensive analysis spans 26 SoTA models
  from 7 major families across varying scales (3-671B parameters) to validate the
  quality of generated evaluations through rigorous algorithmic checks (e.g.,
  citation grounding) and human assessments. We release the YourBench library,
  the Tempora-0325 dataset, 150k+ question answer pairs based on Tempora and all
  evaluation and inference traces to facilitate reproducible research and empower
  the community to generate bespoke benchmarks on demand, fostering more relevant
  and trustworthy LLM evaluation.
paper_ai_summary: YourBench introduces an automated framework for generating domain-specific LLM benchmarks dynamically and cost-effectively from user documents, ensuring reliable and up-to-date evaluations.
paper_ai_keywords: large language models YourBench dynamic benchmarks automated generation domain-tailored benchmarks Tempora-0325 MMLU subsets Tempora-0325 dataset citation grounding SoTA models parameter-efficient fine-tuning reproducible research