title: ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended
    Capabilities
summary: Traditional fixed test sets fall short in evaluating open-ended capabilities
  of foundation models. To address this, we propose ONEBench(OpeN-Ended
  Benchmarking), a new testing paradigm that consolidates individual evaluation
  datasets into a unified, ever-expanding sample pool. ONEBench allows users to
  generate custom, open-ended evaluation benchmarks from this pool, corresponding
  to specific capabilities of interest. By aggregating samples across test sets,
  ONEBench enables the assessment of diverse capabilities beyond those covered by
  the original test sets, while mitigating overfitting and dataset bias. Most
  importantly, it frames model evaluation as a collective process of selecting
  and aggregating sample-level tests.
    The shift from task-specific benchmarks to ONEBench introduces two
  challenges: (1)heterogeneity and (2)incompleteness. Heterogeneity refers to the
  aggregation over diverse metrics, while incompleteness describes comparing
  models evaluated on different data subsets. To address these challenges, we
  explore algorithms to aggregate sparse measurements into reliable model scores.
  Our aggregation algorithm ensures identifiability(asymptotically recovering
  ground-truth scores) and rapid convergence, enabling accurate model ranking
  with less data. On homogenous datasets, we show our aggregation algorithm
  provides rankings that highly correlate with those produced by average scores.
  We also demonstrate robustness to ~95% of measurements missing, reducing
  evaluation cost by up to 20x with little-to-no change in model rankings. We
  introduce ONEBench-LLM for language models and ONEBench-LMM for vision-language
  models, unifying evaluations across these domains. Overall, we present a
  technique for open-ended evaluation, which can aggregate over incomplete,
  heterogeneous sample-level measurements to continually grow a benchmark
  alongside the rapidly developing foundation models.
paper_authors._id: 675b59786409aa5995363472
paper_authors.name: Adhiraj Ghosh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:22:49.556Z
paper_authors._id: 6321b4417bb41a713da9a1f4
paper_authors.avatarUrl: /avatars/f2cde57366791eabee12d456c833baae.svg
paper_authors.fullname: Adhiraj Ghosh
paper_authors.type: user
paper_authors.user: adhiraj1998
paper_authors._id: 675b59786409aa5995363473
paper_authors.name: Sebastian Dziadzio
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:00:58.634Z
paper_authors._id: 656202582c671784e2cf1440
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4cpkcwGn3Z6oKquShOXqC.jpeg
paper_authors.fullname: Sebastian Dziadzio
paper_authors.type: user
paper_authors.user: sbdzdz
paper_authors._id: 675b59786409aa5995363474
paper_authors.name: Ameya Prabhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:01:10.379Z
paper_authors._id: 6464a0d41683d3c81f51924a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6464a0d41683d3c81f51924a/s7yYVwfUB4WOhVFJS6A6T.jpeg
paper_authors.fullname: Ameya Prabhu
paper_authors.type: user
paper_authors.user: AmeyaPrabhu
paper_authors._id: 675b59786409aa5995363475
paper_authors.name: Vishaal Udandarao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:01:16.873Z
paper_authors._id: 6304da46ce6b12280b1bd575
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6304da46ce6b12280b1bd575/V96ocKW4HOoysAGxuAH1X.jpeg
paper_authors.fullname: Vishaal Udandarao
paper_authors.type: user
paper_authors.user: vishaal27
paper_authors._id: 675b59786409aa5995363476
paper_authors.name: Samuel Albanie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:01:22.833Z
paper_authors._id: 62f3efefd6ba2ee26651f44a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660153837083-noauth.png
paper_authors.fullname: Samuel Albanie
paper_authors.type: user
paper_authors.user: albanie
paper_authors._id: 675b59786409aa5995363477
paper_authors.name: Matthias Bethge
paper_title: ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended
    Capabilities
paper_summary: Traditional fixed test sets fall short in evaluating open-ended capabilities
  of foundation models. To address this, we propose ONEBench(OpeN-Ended
  Benchmarking), a new testing paradigm that consolidates individual evaluation
  datasets into a unified, ever-expanding sample pool. ONEBench allows users to
  generate custom, open-ended evaluation benchmarks from this pool, corresponding
  to specific capabilities of interest. By aggregating samples across test sets,
  ONEBench enables the assessment of diverse capabilities beyond those covered by
  the original test sets, while mitigating overfitting and dataset bias. Most
  importantly, it frames model evaluation as a collective process of selecting
  and aggregating sample-level tests.
    The shift from task-specific benchmarks to ONEBench introduces two
  challenges: (1)heterogeneity and (2)incompleteness. Heterogeneity refers to the
  aggregation over diverse metrics, while incompleteness describes comparing
  models evaluated on different data subsets. To address these challenges, we
  explore algorithms to aggregate sparse measurements into reliable model scores.
  Our aggregation algorithm ensures identifiability(asymptotically recovering
  ground-truth scores) and rapid convergence, enabling accurate model ranking
  with less data. On homogenous datasets, we show our aggregation algorithm
  provides rankings that highly correlate with those produced by average scores.
  We also demonstrate robustness to ~95% of measurements missing, reducing
  evaluation cost by up to 20x with little-to-no change in model rankings. We
  introduce ONEBench-LLM for language models and ONEBench-LMM for vision-language
  models, unifying evaluations across these domains. Overall, we present a
  technique for open-ended evaluation, which can aggregate over incomplete,
  heterogeneous sample-level measurements to continually grow a benchmark
  alongside the rapidly developing foundation models.
paper_ai_keywords: ONEBench OpeN-Ended Benchmarking foundation models open-ended capabilities evaluation benchmarks custom evaluation benchmarks model evaluation sample-level tests task-specific benchmarks heterogeneity incompleteness sparse measurements model scores identifiability asymptotically recovering ground-truth scores rapid convergence model ranking average scores ONEBench-LLM ONEBench-LMM language models vision-language models open-ended evaluation heterogeneous sample-level measurements