title: CoverBench: A Challenging Benchmark for Complex Claim Verification
summary: There is a growing line of research on verifying the correctness of language
  models' outputs. At the same time, LMs are being used to tackle complex queries
  that require reasoning. We introduce CoverBench, a challenging benchmark
  focused on verifying LM outputs in complex reasoning settings. Datasets that
  can be used for this purpose are often designed for other complex reasoning
  tasks (e.g., QA) targeting specific use-cases (e.g., financial tables),
  requiring transformations, negative sampling and selection of hard examples to
  collect such a benchmark. CoverBench provides a diversified evaluation for
  complex claim verification in a variety of domains, types of reasoning,
  relatively long inputs, and a variety of standardizations, such as multiple
  representations for tables where available, and a consistent schema. We
  manually vet the data for quality to ensure low levels of label noise. Finally,
  we report a variety of competitive baseline results to show CoverBench is
  challenging and has very significant headroom. The data is available at
  https://huggingface.co/datasets/google/coverbench .
paper_authors._id: 66b2d3ffe8d4c455ad38854f
paper_authors.name: Alon Jacovi
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-08-07T05:40:46.151Z
paper_authors._id: 6596c90dfe0f9826a63654d0
paper_authors.avatarUrl: /avatars/e616fbdff1a345ad22081f5cd019329a.svg
paper_authors.fullname: Alon Jacovi
paper_authors.type: user
paper_authors.user: alonjacovi
paper_authors._id: 66b2d3ffe8d4c455ad388550
paper_authors.name: Moran Ambar
paper_authors._id: 66b2d3ffe8d4c455ad388551
paper_authors.name: Eyal Ben-David
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:25:51.985Z
paper_authors._id: 6645fc650e6706053171ce51
paper_authors.avatarUrl: /avatars/54b03ac6939d4b8943606b12b979ce52.svg
paper_authors.fullname: Eyal Ben-David
paper_authors.type: user
paper_authors.user: eyalbd
paper_authors._id: 66b2d3ffe8d4c455ad388552
paper_authors.name: Uri Shaham
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:26:00.762Z
paper_authors._id: 61265535ed8ae1abb08326d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629996654743-61265535ed8ae1abb08326d4.jpeg
paper_authors.fullname: Uri Shaham
paper_authors.type: user
paper_authors.user: Uri
paper_authors._id: 66b2d3ffe8d4c455ad388553
paper_authors.name: Amir Feder
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:26:06.294Z
paper_authors._id: 63373a1fb575b33cf0200c19
paper_authors.avatarUrl: /avatars/f348182455c1dc09d87b8f8ce5824bec.svg
paper_authors.fullname: Amir Feder
paper_authors.type: user
paper_authors.user: amirfeder
paper_authors._id: 66b2d3ffe8d4c455ad388554
paper_authors.name: Mor Geva
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:26:12.244Z
paper_authors._id: 610b729f9da682cd54ad9adf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1628140189042-noauth.jpeg
paper_authors.fullname: Mor Geva
paper_authors.type: user
paper_authors.user: mega
paper_authors._id: 66b2d3ffe8d4c455ad388555
paper_authors.name: Dror Marcus
paper_authors._id: 66b2d3ffe8d4c455ad388556
paper_authors.name: Avi Caciularu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T12:26:24.882Z
paper_authors._id: 6033d0681f993496bc14d9eb
paper_authors.avatarUrl: /avatars/1c0dfc1fe7b62a78b76ec4f9e6b40b24.svg
paper_authors.fullname: Avi Caciularu
paper_authors.type: user
paper_authors.user: codevan
paper_title: CoverBench: A Challenging Benchmark for Complex Claim Verification
paper_summary: There is a growing line of research on verifying the correctness of language
  models' outputs. At the same time, LMs are being used to tackle complex queries
  that require reasoning. We introduce CoverBench, a challenging benchmark
  focused on verifying LM outputs in complex reasoning settings. Datasets that
  can be used for this purpose are often designed for other complex reasoning
  tasks (e.g., QA) targeting specific use-cases (e.g., financial tables),
  requiring transformations, negative sampling and selection of hard examples to
  collect such a benchmark. CoverBench provides a diversified evaluation for
  complex claim verification in a variety of domains, types of reasoning,
  relatively long inputs, and a variety of standardizations, such as multiple
  representations for tables where available, and a consistent schema. We
  manually vet the data for quality to ensure low levels of label noise. Finally,
  we report a variety of competitive baseline results to show CoverBench is
  challenging and has very significant headroom. The data is available at
  https://huggingface.co/datasets/google/coverbench .
paper_ai_summary: CoverBench is a challenging benchmark for verifying the correctness of language model outputs in complex reasoning settings, providing diverse evaluations across domains and reasoning types.
paper_ai_keywords: language models claim verification complex reasoning benchmark datasets QA financial tables negative sampling manual vetting label noise baseline results huggingface