title: BIG-Bench Extra Hard
summary: Large language models (LLMs) are increasingly deployed in everyday
  applications, demanding robust general reasoning capabilities and diverse
  reasoning skillset. However, current LLM reasoning benchmarks predominantly
  focus on mathematical and coding abilities, leaving a gap in evaluating broader
  reasoning proficiencies. One particular exception is the BIG-Bench dataset,
  which has served as a crucial benchmark for evaluating the general reasoning
  capabilities of LLMs, thanks to its diverse set of challenging tasks that
  allowed for a comprehensive assessment of general reasoning across various
  skills within a unified framework. However, recent advances in LLMs have led to
  saturation on BIG-Bench, and its harder version BIG-Bench Hard (BBH).
  State-of-the-art models achieve near-perfect scores on many tasks in BBH, thus
  diminishing its utility. To address this limitation, we introduce BIG-Bench
  Extra Hard (BBEH), a new benchmark designed to push the boundaries of LLM
  reasoning evaluation. BBEH replaces each task in BBH with a novel task that
  probes a similar reasoning capability but exhibits significantly increased
  difficulty. We evaluate various models on BBEH and observe a (harmonic) average
  accuracy of 9.8\% for the best general-purpose model and 44.8\% for the best
  reasoning-specialized model, indicating substantial room for improvement and
  highlighting the ongoing challenge of achieving robust general reasoning in
  LLMs. We release BBEH publicly at: https://github.com/google-deepmind/bbeh.
paper_authors._id: 67c01747e8c7d56a8e0cbdc3
paper_authors.name: Mehran Kazemi
paper_authors._id: 67c01747e8c7d56a8e0cbdc4
paper_authors.name: Bahare Fatemi
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-27T07:42:00.525Z
paper_authors._id: 654e97ef5da3196a78409341
paper_authors.avatarUrl: /avatars/1a5ea7351ca21960891cf9721b9f4667.svg
paper_authors.fullname: Bahare Fatemi
paper_authors.type: user
paper_authors.user: baharefatemi
paper_authors._id: 67c01747e8c7d56a8e0cbdc5
paper_authors.name: Hritik Bansal
paper_authors._id: 67c01747e8c7d56a8e0cbdc6
paper_authors.name: John Palowitch
paper_authors._id: 67c01747e8c7d56a8e0cbdc7
paper_authors.name: Chrysovalantis Anastasiou
paper_authors._id: 67c01747e8c7d56a8e0cbdc8
paper_authors.name: Sanket Vaibhav Mehta
paper_authors._id: 67c01747e8c7d56a8e0cbdc9
paper_authors.name: Lalit K. Jain
paper_authors._id: 67c01747e8c7d56a8e0cbdca
paper_authors.name: Virginia Aglietti
paper_authors._id: 67c01747e8c7d56a8e0cbdcb
paper_authors.name: Disha Jindal
paper_authors._id: 67c01747e8c7d56a8e0cbdcc
paper_authors.name: Peter Chen
paper_authors._id: 67c01747e8c7d56a8e0cbdcd
paper_authors.name: Nishanth Dikkala
paper_authors._id: 67c01747e8c7d56a8e0cbdce
paper_authors.name: Gladys Tyen
paper_authors._id: 67c01747e8c7d56a8e0cbdcf
paper_authors.name: Xin Liu
paper_authors._id: 67c01747e8c7d56a8e0cbdd0
paper_authors.name: Uri Shalit
paper_authors._id: 67c01747e8c7d56a8e0cbdd1
paper_authors.name: Silvia Chiappa
paper_authors._id: 67c01747e8c7d56a8e0cbdd2
paper_authors.name: Kate Olszewska
paper_authors._id: 67c01747e8c7d56a8e0cbdd3
paper_authors.name: Yi Tay
paper_authors._id: 67c01747e8c7d56a8e0cbdd4
paper_authors.name: Vinh Q. Tran
paper_authors._id: 67c01747e8c7d56a8e0cbdd5
paper_authors.name: Quoc V. Le
paper_authors._id: 67c01747e8c7d56a8e0cbdd6
paper_authors.name: Orhan Firat
paper_title: BIG-Bench Extra Hard
paper_summary: Large language models (LLMs) are increasingly deployed in everyday
  applications, demanding robust general reasoning capabilities and diverse
  reasoning skillset. However, current LLM reasoning benchmarks predominantly
  focus on mathematical and coding abilities, leaving a gap in evaluating broader
  reasoning proficiencies. One particular exception is the BIG-Bench dataset,
  which has served as a crucial benchmark for evaluating the general reasoning
  capabilities of LLMs, thanks to its diverse set of challenging tasks that
  allowed for a comprehensive assessment of general reasoning across various
  skills within a unified framework. However, recent advances in LLMs have led to
  saturation on BIG-Bench, and its harder version BIG-Bench Hard (BBH).
  State-of-the-art models achieve near-perfect scores on many tasks in BBH, thus
  diminishing its utility. To address this limitation, we introduce BIG-Bench
  Extra Hard (BBEH), a new benchmark designed to push the boundaries of LLM
  reasoning evaluation. BBEH replaces each task in BBH with a novel task that
  probes a similar reasoning capability but exhibits significantly increased
  difficulty. We evaluate various models on BBEH and observe a (harmonic) average
  accuracy of 9.8\% for the best general-purpose model and 44.8\% for the best
  reasoning-specialized model, indicating substantial room for improvement and
  highlighting the ongoing challenge of achieving robust general reasoning in
  LLMs. We release BBEH publicly at: https://github.com/google-deepmind/bbeh.
paper_ai_summary: BIG-Bench Extra Hard (BBEH) introduces new tasks to challenge LLMs, showing significant room for improvement in general reasoning capabilities beyond current benchmarks.
paper_ai_keywords: BIG-Bench BIG-Bench Hard (BBH) BIG-Bench Extra Hard (BBEH) large language models (LLMs) reasoning benchmarks general reasoning capabilities reasoning skillset mathematical abilities coding abilities diverse set of challenging tasks comprehensive assessment general reasoning unified framework recent advances saturation harmonic average accuracy general-purpose model reasoning-specialized model