SciBench: Evaluating College-Level Scientific Problem-Solving Abilities
  of Large Language Models
Recent advances in large language models (LLMs) have demonstrated notable
progress on many mathematical benchmarks. However, most of these benchmarks
only feature problems grounded in junior and senior high school subjects,
contain only multiple-choice questions, and are confined to a limited scope of
elementary arithmetic operations. To address these issues, this paper
introduces an expansive benchmark suite SciBench that aims to systematically
examine the reasoning capabilities required for complex scientific problem
solving. SciBench contains two carefully curated datasets: an open set
featuring a range of collegiate-level scientific problems drawn from
mathematics, chemistry, and physics textbooks, and a closed set comprising
problems from undergraduate-level exams in computer science and mathematics.
Based on the two datasets, we conduct an in-depth benchmark study of two
representative LLMs with various prompting strategies. The results reveal that
current LLMs fall short of delivering satisfactory performance, with an overall
score of merely 35.80%. Furthermore, through a detailed user study, we
categorize the errors made by LLMs into ten problem-solving abilities. Our
analysis indicates that no single prompting strategy significantly outperforms
others and some strategies that demonstrate improvements in certain
problem-solving skills result in declines in other skills. We envision that
SciBench will catalyze further developments in the reasoning abilities of LLMs,
thereby ultimately contributing to scientific research and discovery.
64ba0f0a5d5cc69c0470f40f
Xiaoxuan Wang
claimed_verified
2025-04-01T07:49:11.240Z
64ba5946c0f19c9025665a3c
/avatars/bb148094ce52f1f385d30968dc22e0e6.svg
Xiaoxuan Wang
user
xw27
64ba0f0a5d5cc69c0470f410
Ziniu Hu
64ba0f0a5d5cc69c0470f411
Pan Lu
claimed_verified
2025-02-19T09:08:47.930Z
60f5f68fa7fd83d025749234
https://cdn-avatars.huggingface.co/v1/production/uploads/60f5f68fa7fd83d025749234/gCeJAZfzaANAcEvI6v5-P.jpeg
Pan Lu
user
lupantech
64ba0f0a5d5cc69c0470f412
Yanqiao Zhu
claimed_verified
2024-07-02T07:40:31.839Z
65dda5295eea01ee95e21f64
/avatars/f935e7db6038661bef2305b7aeda8c9e.svg
Yanqiao Zhu
user
sxkdz
64ba0f0a5d5cc69c0470f413
Jieyu Zhang
admin_assigned
2023-07-21T08:14:07.915Z
640131b08ba76abe4b71b5d0
/avatars/2288b96a9a0ae8f584768f54e098def1.svg
Jieyu Zhang
user
jieyuz2
64ba0f0a5d5cc69c0470f414
Satyen Subramaniam
64ba0f0a5d5cc69c0470f415
Arjun R. Loomba
64ba0f0a5d5cc69c0470f416
Shichang Zhang
admin_assigned
2023-07-21T08:14:44.897Z
64b6d98861dc301f1326341a
/avatars/14df7497a1a982894f5889903793773f.svg
Shichang Zhang
user
shichangzh
64ba0f0a5d5cc69c0470f417
Yizhou Sun
64ba0f0a5d5cc69c0470f418
Wei Wang
SciBench: Evaluating College-Level Scientific Problem-Solving Abilities
  of Large Language Models
Recent advances in large language models (LLMs) have demonstrated notable
progress on many mathematical benchmarks. However, most of these benchmarks
only feature problems grounded in junior and senior high school subjects,
contain only multiple-choice questions, and are confined to a limited scope of
elementary arithmetic operations. To address these issues, this paper
introduces an expansive benchmark suite SciBench that aims to systematically
examine the reasoning capabilities required for complex scientific problem
solving. SciBench contains two carefully curated datasets: an open set
featuring a range of collegiate-level scientific problems drawn from
mathematics, chemistry, and physics textbooks, and a closed set comprising
problems from undergraduate-level exams in computer science and mathematics.
Based on the two datasets, we conduct an in-depth benchmark study of two
representative LLMs with various prompting strategies. The results reveal that
current LLMs fall short of delivering satisfactory performance, with an overall
score of merely 35.80%. Furthermore, through a detailed user study, we
categorize the errors made by LLMs into ten problem-solving abilities. Our
analysis indicates that no single prompting strategy significantly outperforms
others and some strategies that demonstrate improvements in certain
problem-solving skills result in declines in other skills. We envision that
SciBench will catalyze further developments in the reasoning abilities of LLMs,
thereby ultimately contributing to scientific research and discovery.
SciBench introduces an extensive benchmark suite to evaluate the reasoning capabilities of large language models on complex scientific problems, revealing areas for improvement in these models' performance.
large language models LLMs mathematical benchmarks junior high school subjects senior high school subjects multiple-choice questions elementary arithmetic operations collegiate-level scientific problems undergraduate-level exams computer science mathematics chemistry physics benchmark study prompting strategies problem-solving abilities