ScienceAgentBench: Toward Rigorous Assessment of Language Agents for
  Data-Driven Scientific Discovery
The advancements of language language models (LLMs) have piqued growing
interest in developing LLM-based language agents to automate scientific
discovery end-to-end, which has sparked both excitement and skepticism about
the true capabilities of such agents. In this work, we argue that for an agent
to fully automate scientific discovery, it must be able to complete all
essential tasks in the workflow. Thus, we call for rigorous assessment of
agents on individual tasks in a scientific workflow before making bold claims
on end-to-end automation. To this end, we present ScienceAgentBench, a new
benchmark for evaluating language agents for data-driven scientific discovery.
To ensure the scientific authenticity and real-world relevance of our
benchmark, we extract 102 tasks from 44 peer-reviewed publications in four
disciplines and engage nine subject matter experts to validate them. We unify
the target output for every task to a self-contained Python program file and
employ an array of evaluation metrics to examine the generated programs,
execution results, and costs. Each task goes through multiple rounds of manual
validation by annotators and subject matter experts to ensure its annotation
quality and scientific plausibility. We also propose two effective strategies
to mitigate data contamination concerns. Using our benchmark, we evaluate five
open-weight and proprietary LLMs, each with three frameworks: direct prompting,
OpenHands, and self-debug. Given three attempts for each task, the
best-performing agent can only solve 32.4% of the tasks independently and 34.3%
with expert-provided knowledge. These results underscore the limited capacities
of current language agents in generating code for data-driven discovery, let
alone end-to-end automation for scientific research.
670541503262801fdbc1f7bd
Ziru Chen
claimed_verified
2024-10-09T07:37:51.542Z
6494bbbb185ae221545a0ef6
https://cdn-avatars.huggingface.co/v1/production/uploads/6494bbbb185ae221545a0ef6/oA6HXlD0KnNfttYf2f_qD.jpeg
Ziru Chen
user
ronch99
670541503262801fdbc1f7be
Shijie Chen
670541503262801fdbc1f7bf
Yuting Ning
claimed_verified
2024-10-14T07:48:45.268Z
65ace92f64c9b93eca5c2bce
https://cdn-avatars.huggingface.co/v1/production/uploads/65ace92f64c9b93eca5c2bce/pG0JRXH-8zEy0IoaEnMNw.jpeg
Yuting Ning
user
nnnyt
670541503262801fdbc1f7c0
Qianheng Zhang
670541503262801fdbc1f7c1
Boshi Wang
670541503262801fdbc1f7c2
Botao Yu
670541503262801fdbc1f7c3
Yifei Li
670541503262801fdbc1f7c4
Zeyi Liao
claimed_verified
2025-06-02T07:57:19.285Z
632ed3e534c562cb1d328fa7
/avatars/e29e47e734740787ea40c729eec146b5.svg
zeyi liao
user
lzy337
670541503262801fdbc1f7c5
Chen Wei
670541503262801fdbc1f7c6
Zitong Lu
670541503262801fdbc1f7c7
Vishal Dey
670541503262801fdbc1f7c8
Mingyi Xue
670541503262801fdbc1f7c9
Frazier N. Baker
admin_assigned
2024-10-08T15:41:02.694Z
6557abbf03e1f0503bfbfe77
/avatars/477ecbd68f00353ceee77c5f1832ef73.svg
Frazier Baker
user
frazierbaker
670541503262801fdbc1f7ca
Benjamin Burns
670541503262801fdbc1f7cb
Daniel Adu-Ampratwum
670541503262801fdbc1f7cc
Xuhui Huang
670541503262801fdbc1f7cd
Xia Ning
670541503262801fdbc1f7ce
Song Gao
670541503262801fdbc1f7cf
Yu Su
claimed_verified
2024-10-14T19:21:52.229Z
6477a323dbc2a416f8b852b3
https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
Yu Su
user
ysu-nlp
670541503262801fdbc1f7d0
Huan Sun
ScienceAgentBench: Toward Rigorous Assessment of Language Agents for
  Data-Driven Scientific Discovery
The advancements of language language models (LLMs) have piqued growing
interest in developing LLM-based language agents to automate scientific
discovery end-to-end, which has sparked both excitement and skepticism about
the true capabilities of such agents. In this work, we argue that for an agent
to fully automate scientific discovery, it must be able to complete all
essential tasks in the workflow. Thus, we call for rigorous assessment of
agents on individual tasks in a scientific workflow before making bold claims
on end-to-end automation. To this end, we present ScienceAgentBench, a new
benchmark for evaluating language agents for data-driven scientific discovery.
To ensure the scientific authenticity and real-world relevance of our
benchmark, we extract 102 tasks from 44 peer-reviewed publications in four
disciplines and engage nine subject matter experts to validate them. We unify
the target output for every task to a self-contained Python program file and
employ an array of evaluation metrics to examine the generated programs,
execution results, and costs. Each task goes through multiple rounds of manual
validation by annotators and subject matter experts to ensure its annotation
quality and scientific plausibility. We also propose two effective strategies
to mitigate data contamination concerns. Using our benchmark, we evaluate five
open-weight and proprietary LLMs, each with three frameworks: direct prompting,
OpenHands, and self-debug. Given three attempts for each task, the
best-performing agent can only solve 32.4% of the tasks independently and 34.3%
with expert-provided knowledge. These results underscore the limited capacities
of current language agents in generating code for data-driven discovery, let
alone end-to-end automation for scientific research.
ScienceAgentBench evaluates language agents for scientific discovery tasks, showing limited capability in generating code for data-driven research.
language language models (LLMs) language agents scientific discovery end-to-end automation ScienceAgentBench peer-reviewed publications Python program file evaluation metrics direct prompting OpenHands self-debug data contamination