ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended
  Capabilities
Traditional fixed test sets fall short in evaluating open-ended capabilities
of foundation models. To address this, we propose ONEBench(OpeN-Ended
Benchmarking), a new testing paradigm that consolidates individual evaluation
datasets into a unified, ever-expanding sample pool. ONEBench allows users to
generate custom, open-ended evaluation benchmarks from this pool, corresponding
to specific capabilities of interest. By aggregating samples across test sets,
ONEBench enables the assessment of diverse capabilities beyond those covered by
the original test sets, while mitigating overfitting and dataset bias. Most
importantly, it frames model evaluation as a collective process of selecting
and aggregating sample-level tests.
  The shift from task-specific benchmarks to ONEBench introduces two
challenges: (1)heterogeneity and (2)incompleteness. Heterogeneity refers to the
aggregation over diverse metrics, while incompleteness describes comparing
models evaluated on different data subsets. To address these challenges, we
explore algorithms to aggregate sparse measurements into reliable model scores.
Our aggregation algorithm ensures identifiability(asymptotically recovering
ground-truth scores) and rapid convergence, enabling accurate model ranking
with less data. On homogenous datasets, we show our aggregation algorithm
provides rankings that highly correlate with those produced by average scores.
We also demonstrate robustness to ~95% of measurements missing, reducing
evaluation cost by up to 20x with little-to-no change in model rankings. We
introduce ONEBench-LLM for language models and ONEBench-LMM for vision-language
models, unifying evaluations across these domains. Overall, we present a
technique for open-ended evaluation, which can aggregate over incomplete,
heterogeneous sample-level measurements to continually grow a benchmark
alongside the rapidly developing foundation models.
675b59786409aa5995363472
Adhiraj Ghosh
claimed_verified
2024-12-13T15:22:49.556Z
6321b4417bb41a713da9a1f4
/avatars/f2cde57366791eabee12d456c833baae.svg
Adhiraj Ghosh
user
adhiraj1998
675b59786409aa5995363473
Sebastian Dziadzio
admin_assigned
2024-12-16T11:00:58.634Z
656202582c671784e2cf1440
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4cpkcwGn3Z6oKquShOXqC.jpeg
Sebastian Dziadzio
user
sbdzdz
675b59786409aa5995363474
Ameya Prabhu
admin_assigned
2024-12-16T11:01:10.379Z
6464a0d41683d3c81f51924a
https://cdn-avatars.huggingface.co/v1/production/uploads/6464a0d41683d3c81f51924a/s7yYVwfUB4WOhVFJS6A6T.jpeg
Ameya Prabhu
user
AmeyaPrabhu
675b59786409aa5995363475
Vishaal Udandarao
admin_assigned
2024-12-16T11:01:16.873Z
6304da46ce6b12280b1bd575
https://cdn-avatars.huggingface.co/v1/production/uploads/6304da46ce6b12280b1bd575/V96ocKW4HOoysAGxuAH1X.jpeg
Vishaal Udandarao
user
vishaal27
675b59786409aa5995363476
Samuel Albanie
admin_assigned
2024-12-16T11:01:22.833Z
62f3efefd6ba2ee26651f44a
https://cdn-avatars.huggingface.co/v1/production/uploads/1660153837083-noauth.png
Samuel Albanie
user
albanie
675b59786409aa5995363477
Matthias Bethge
ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended
  Capabilities
Traditional fixed test sets fall short in evaluating open-ended capabilities
of foundation models. To address this, we propose ONEBench(OpeN-Ended
Benchmarking), a new testing paradigm that consolidates individual evaluation
datasets into a unified, ever-expanding sample pool. ONEBench allows users to
generate custom, open-ended evaluation benchmarks from this pool, corresponding
to specific capabilities of interest. By aggregating samples across test sets,
ONEBench enables the assessment of diverse capabilities beyond those covered by
the original test sets, while mitigating overfitting and dataset bias. Most
importantly, it frames model evaluation as a collective process of selecting
and aggregating sample-level tests.
  The shift from task-specific benchmarks to ONEBench introduces two
challenges: (1)heterogeneity and (2)incompleteness. Heterogeneity refers to the
aggregation over diverse metrics, while incompleteness describes comparing
models evaluated on different data subsets. To address these challenges, we
explore algorithms to aggregate sparse measurements into reliable model scores.
Our aggregation algorithm ensures identifiability(asymptotically recovering
ground-truth scores) and rapid convergence, enabling accurate model ranking
with less data. On homogenous datasets, we show our aggregation algorithm
provides rankings that highly correlate with those produced by average scores.
We also demonstrate robustness to ~95% of measurements missing, reducing
evaluation cost by up to 20x with little-to-no change in model rankings. We
introduce ONEBench-LLM for language models and ONEBench-LMM for vision-language
models, unifying evaluations across these domains. Overall, we present a
technique for open-ended evaluation, which can aggregate over incomplete,
heterogeneous sample-level measurements to continually grow a benchmark
alongside the rapidly developing foundation models.
ONEBench consolidates and evaluates foundation models using an expanding pool of heterogeneous samples, addressing overfitting and dataset bias with aggregation algorithms that enable accurate and efficient model ranking.
ONEBench open-ended benchmarking sample-level tests model evaluation aggregation algorithm identifiability rapid convergence ONEBench-LLM ONEBench-LMM vision-language models language models