title: SciBench: Evaluating College-Level Scientific Problem-Solving Abilities
    of Large Language Models
summary: Recent advances in large language models (LLMs) have demonstrated notable
  progress on many mathematical benchmarks. However, most of these benchmarks
  only feature problems grounded in junior and senior high school subjects,
  contain only multiple-choice questions, and are confined to a limited scope of
  elementary arithmetic operations. To address these issues, this paper
  introduces an expansive benchmark suite SciBench that aims to systematically
  examine the reasoning capabilities required for complex scientific problem
  solving. SciBench contains two carefully curated datasets: an open set
  featuring a range of collegiate-level scientific problems drawn from
  mathematics, chemistry, and physics textbooks, and a closed set comprising
  problems from undergraduate-level exams in computer science and mathematics.
  Based on the two datasets, we conduct an in-depth benchmark study of two
  representative LLMs with various prompting strategies. The results reveal that
  current LLMs fall short of delivering satisfactory performance, with an overall
  score of merely 35.80%. Furthermore, through a detailed user study, we
  categorize the errors made by LLMs into ten problem-solving abilities. Our
  analysis indicates that no single prompting strategy significantly outperforms
  others and some strategies that demonstrate improvements in certain
  problem-solving skills result in declines in other skills. We envision that
  SciBench will catalyze further developments in the reasoning abilities of LLMs,
  thereby ultimately contributing to scientific research and discovery.
paper_authors._id: 64ba0f0a5d5cc69c0470f40f
paper_authors.name: Xiaoxuan Wang
paper_authors._id: 64ba0f0a5d5cc69c0470f410
paper_authors.name: Ziniu Hu
paper_authors._id: 64ba0f0a5d5cc69c0470f411
paper_authors.name: Pan Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:08:47.930Z
paper_authors._id: 60f5f68fa7fd83d025749234
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60f5f68fa7fd83d025749234/gCeJAZfzaANAcEvI6v5-P.jpeg
paper_authors.fullname: Pan Lu
paper_authors.type: user
paper_authors.user: lupantech
paper_authors._id: 64ba0f0a5d5cc69c0470f412
paper_authors.name: Yanqiao Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T07:40:31.839Z
paper_authors._id: 65dda5295eea01ee95e21f64
paper_authors.avatarUrl: /avatars/f935e7db6038661bef2305b7aeda8c9e.svg
paper_authors.fullname: Yanqiao Zhu
paper_authors.type: user
paper_authors.user: sxkdz
paper_authors._id: 64ba0f0a5d5cc69c0470f413
paper_authors.name: Jieyu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-21T08:14:07.915Z
paper_authors._id: 640131b08ba76abe4b71b5d0
paper_authors.avatarUrl: /avatars/2288b96a9a0ae8f584768f54e098def1.svg
paper_authors.fullname: Jieyu Zhang
paper_authors.type: user
paper_authors.user: jieyuz2
paper_authors._id: 64ba0f0a5d5cc69c0470f414
paper_authors.name: Satyen Subramaniam
paper_authors._id: 64ba0f0a5d5cc69c0470f415
paper_authors.name: Arjun R. Loomba
paper_authors._id: 64ba0f0a5d5cc69c0470f416
paper_authors.name: Shichang Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-21T08:14:44.897Z
paper_authors._id: 64b6d98861dc301f1326341a
paper_authors.avatarUrl: /avatars/14df7497a1a982894f5889903793773f.svg
paper_authors.fullname: Shichang Zhang
paper_authors.type: user
paper_authors.user: shichangzh
paper_authors._id: 64ba0f0a5d5cc69c0470f417
paper_authors.name: Yizhou Sun
paper_authors._id: 64ba0f0a5d5cc69c0470f418
paper_authors.name: Wei Wang
paper_title: SciBench: Evaluating College-Level Scientific Problem-Solving Abilities
    of Large Language Models
paper_summary: Recent advances in large language models (LLMs) have demonstrated notable
  progress on many mathematical benchmarks. However, most of these benchmarks
  only feature problems grounded in junior and senior high school subjects,
  contain only multiple-choice questions, and are confined to a limited scope of
  elementary arithmetic operations. To address these issues, this paper
  introduces an expansive benchmark suite SciBench that aims to systematically
  examine the reasoning capabilities required for complex scientific problem
  solving. SciBench contains two carefully curated datasets: an open set
  featuring a range of collegiate-level scientific problems drawn from
  mathematics, chemistry, and physics textbooks, and a closed set comprising
  problems from undergraduate-level exams in computer science and mathematics.
  Based on the two datasets, we conduct an in-depth benchmark study of two
  representative LLMs with various prompting strategies. The results reveal that
  current LLMs fall short of delivering satisfactory performance, with an overall
  score of merely 35.80%. Furthermore, through a detailed user study, we
  categorize the errors made by LLMs into ten problem-solving abilities. Our
  analysis indicates that no single prompting strategy significantly outperforms
  others and some strategies that demonstrate improvements in certain
  problem-solving skills result in declines in other skills. We envision that
  SciBench will catalyze further developments in the reasoning abilities of LLMs,
  thereby ultimately contributing to scientific research and discovery.