CoverBench: A Challenging Benchmark for Complex Claim Verification
There is a growing line of research on verifying the correctness of language
models' outputs. At the same time, LMs are being used to tackle complex queries
that require reasoning. We introduce CoverBench, a challenging benchmark
focused on verifying LM outputs in complex reasoning settings. Datasets that
can be used for this purpose are often designed for other complex reasoning
tasks (e.g., QA) targeting specific use-cases (e.g., financial tables),
requiring transformations, negative sampling and selection of hard examples to
collect such a benchmark. CoverBench provides a diversified evaluation for
complex claim verification in a variety of domains, types of reasoning,
relatively long inputs, and a variety of standardizations, such as multiple
representations for tables where available, and a consistent schema. We
manually vet the data for quality to ensure low levels of label noise. Finally,
we report a variety of competitive baseline results to show CoverBench is
challenging and has very significant headroom. The data is available at
https://huggingface.co/datasets/google/coverbench .
66b2d3ffe8d4c455ad38854f
Alon Jacovi
extracted_confirmed
2024-08-07T05:40:46.151Z
6596c90dfe0f9826a63654d0
/avatars/e616fbdff1a345ad22081f5cd019329a.svg
Alon Jacovi
user
alonjacovi
66b2d3ffe8d4c455ad388550
Moran Ambar
66b2d3ffe8d4c455ad388551
Eyal Ben-David
admin_assigned
2024-08-08T12:25:51.985Z
6645fc650e6706053171ce51
/avatars/54b03ac6939d4b8943606b12b979ce52.svg
Eyal Ben-David
user
eyalbd
66b2d3ffe8d4c455ad388552
Uri Shaham
admin_assigned
2024-08-08T12:26:00.762Z
61265535ed8ae1abb08326d4
https://cdn-avatars.huggingface.co/v1/production/uploads/1629996654743-61265535ed8ae1abb08326d4.jpeg
Uri Shaham
user
Uri
66b2d3ffe8d4c455ad388553
Amir Feder
admin_assigned
2024-08-08T12:26:06.294Z
63373a1fb575b33cf0200c19
/avatars/f348182455c1dc09d87b8f8ce5824bec.svg
Amir Feder
user
amirfeder
66b2d3ffe8d4c455ad388554
Mor Geva
admin_assigned
2024-08-08T12:26:12.244Z
610b729f9da682cd54ad9adf
https://cdn-avatars.huggingface.co/v1/production/uploads/1628140189042-noauth.jpeg
Mor Geva
user
mega
66b2d3ffe8d4c455ad388555
Dror Marcus
66b2d3ffe8d4c455ad388556
Avi Caciularu
admin_assigned
2024-08-08T12:26:24.882Z
6033d0681f993496bc14d9eb
/avatars/1c0dfc1fe7b62a78b76ec4f9e6b40b24.svg
Avi Caciularu
user
codevan
CoverBench: A Challenging Benchmark for Complex Claim Verification
There is a growing line of research on verifying the correctness of language
models' outputs. At the same time, LMs are being used to tackle complex queries
that require reasoning. We introduce CoverBench, a challenging benchmark
focused on verifying LM outputs in complex reasoning settings. Datasets that
can be used for this purpose are often designed for other complex reasoning
tasks (e.g., QA) targeting specific use-cases (e.g., financial tables),
requiring transformations, negative sampling and selection of hard examples to
collect such a benchmark. CoverBench provides a diversified evaluation for
complex claim verification in a variety of domains, types of reasoning,
relatively long inputs, and a variety of standardizations, such as multiple
representations for tables where available, and a consistent schema. We
manually vet the data for quality to ensure low levels of label noise. Finally,
we report a variety of competitive baseline results to show CoverBench is
challenging and has very significant headroom. The data is available at
https://huggingface.co/datasets/google/coverbench .
CoverBench is a challenging benchmark for verifying the correctness of language model outputs in complex reasoning settings, providing diverse evaluations across domains and reasoning types.
language models claim verification complex reasoning benchmark datasets QA financial tables negative sampling manual vetting label noise baseline results huggingface