title: ARB: Advanced Reasoning Benchmark for Large Language Models
summary: Large Language Models (LLMs) have demonstrated remarkable performance on
  various quantitative reasoning and knowledge benchmarks. However, many of these
  benchmarks are losing utility as LLMs get increasingly high scores, despite not
  yet reaching expert performance in these domains. We introduce ARB, a novel
  benchmark composed of advanced reasoning problems in multiple fields. ARB
  presents a more challenging test than prior benchmarks, featuring problems in
  mathematics, physics, biology, chemistry, and law. As a subset of ARB, we
  introduce a challenging set of math and physics problems which require advanced
  symbolic reasoning and domain knowledge. We evaluate recent models such as
  GPT-4 and Claude on ARB and demonstrate that current models score well below
  50% on more demanding tasks. In order to improve both automatic and assisted
  evaluation capabilities, we introduce a rubric-based evaluation approach,
  allowing GPT-4 to score its own intermediate reasoning steps. Further, we
  conduct a human evaluation of the symbolic subset of ARB, finding promising
  agreement between annotators and GPT-4 rubric evaluation scores.
paper_authors._id: 64c0846c6cf8e2b9f041d526
paper_authors.name: Tomohiro Sawada
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-27T08:49:18.370Z
paper_authors._id: 63750b55a17c01a551cf41df
paper_authors.avatarUrl: /avatars/f9d72caa51d2cb12a6e3e6f700ad35d9.svg
paper_authors.fullname: Tom Sawada
paper_authors.type: user
paper_authors.user: tsawada
paper_authors._id: 64c0846c6cf8e2b9f041d527
paper_authors.name: Daniel Paleka
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-26T08:58:46.537Z
paper_authors._id: 62363277cd8e1f47c865190a
paper_authors.avatarUrl: /avatars/fb42d42dedd4dde8df4109883cbe7ea9.svg
paper_authors.fullname: Daniel Paleka
paper_authors.type: user
paper_authors.user: dpaleka
paper_authors._id: 64c0846c6cf8e2b9f041d528
paper_authors.name: Alexander Havrilla
paper_authors._id: 64c0846c6cf8e2b9f041d529
paper_authors.name: Pranav Tadepalli
paper_authors._id: 64c0846c6cf8e2b9f041d52a
paper_authors.name: Paula Vidas
paper_authors._id: 64c0846c6cf8e2b9f041d52b
paper_authors.name: Alexander Kranias
paper_authors._id: 64c0846c6cf8e2b9f041d52c
paper_authors.name: John J. Nay
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-26T08:59:35.647Z
paper_authors._id: 63f2b49f7ddf724fbcc534fd
paper_authors.avatarUrl: /avatars/4076a12ac9423ab276d06334c7e392db.svg
paper_authors.fullname: John Nay
paper_authors.type: user
paper_authors.user: johnjnay
paper_authors._id: 64c0846c6cf8e2b9f041d52d
paper_authors.name: Kshitij Gupta
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-14T08:00:36.366Z
paper_authors._id: 625daa0c943e346492b63d21
paper_authors.avatarUrl: /avatars/860f8e9bff85325096b6f8e15102def6.svg
paper_authors.fullname: Kshitij Gupta
paper_authors.type: user
paper_authors.user: kshitijkg
paper_authors._id: 64c0846c6cf8e2b9f041d52e
paper_authors.name: Aran Komatsuzaki
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-26T09:03:20.918Z
paper_authors._id: 5f35f6fa7e583543386216c9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f35f6fa7e583543386216c9/7GR8PX6ahT-2dZdMZYz7T.jpeg
paper_authors.fullname: Aran Komatsuzaki
paper_authors.type: user
paper_authors.user: AranKomatsuzaki
paper_title: ARB: Advanced Reasoning Benchmark for Large Language Models
paper_summary: Large Language Models (LLMs) have demonstrated remarkable performance on
  various quantitative reasoning and knowledge benchmarks. However, many of these
  benchmarks are losing utility as LLMs get increasingly high scores, despite not
  yet reaching expert performance in these domains. We introduce ARB, a novel
  benchmark composed of advanced reasoning problems in multiple fields. ARB
  presents a more challenging test than prior benchmarks, featuring problems in
  mathematics, physics, biology, chemistry, and law. As a subset of ARB, we
  introduce a challenging set of math and physics problems which require advanced
  symbolic reasoning and domain knowledge. We evaluate recent models such as
  GPT-4 and Claude on ARB and demonstrate that current models score well below
  50% on more demanding tasks. In order to improve both automatic and assisted
  evaluation capabilities, we introduce a rubric-based evaluation approach,
  allowing GPT-4 to score its own intermediate reasoning steps. Further, we
  conduct a human evaluation of the symbolic subset of ARB, finding promising
  agreement between annotators and GPT-4 rubric evaluation scores.
paper_ai_summary: ARB is a novel benchmark featuring advanced reasoning problems across multiple domains, demonstrating that current LLMs score poorly on more challenging tasks and introducing a rubric-based evaluation approach.
paper_ai_keywords: Large Language Models (LLMs) ARB advanced reasoning problems mathematics physics biology chemistry law symbolic reasoning GPT-4 Claude rubric-based evaluation human evaluation symbolic subset