title: ScienceAgentBench: Toward Rigorous Assessment of Language Agents for
    Data-Driven Scientific Discovery
summary: The advancements of language language models (LLMs) have piqued growing
  interest in developing LLM-based language agents to automate scientific
  discovery end-to-end, which has sparked both excitement and skepticism about
  the true capabilities of such agents. In this work, we argue that for an agent
  to fully automate scientific discovery, it must be able to complete all
  essential tasks in the workflow. Thus, we call for rigorous assessment of
  agents on individual tasks in a scientific workflow before making bold claims
  on end-to-end automation. To this end, we present ScienceAgentBench, a new
  benchmark for evaluating language agents for data-driven scientific discovery.
  To ensure the scientific authenticity and real-world relevance of our
  benchmark, we extract 102 tasks from 44 peer-reviewed publications in four
  disciplines and engage nine subject matter experts to validate them. We unify
  the target output for every task to a self-contained Python program file and
  employ an array of evaluation metrics to examine the generated programs,
  execution results, and costs. Each task goes through multiple rounds of manual
  validation by annotators and subject matter experts to ensure its annotation
  quality and scientific plausibility. We also propose two effective strategies
  to mitigate data contamination concerns. Using our benchmark, we evaluate five
  open-weight and proprietary LLMs, each with three frameworks: direct prompting,
  OpenHands, and self-debug. Given three attempts for each task, the
  best-performing agent can only solve 32.4% of the tasks independently and 34.3%
  with expert-provided knowledge. These results underscore the limited capacities
  of current language agents in generating code for data-driven discovery, let
  alone end-to-end automation for scientific research.
paper_authors._id: 670541503262801fdbc1f7bd
paper_authors.name: Ziru Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-09T07:37:51.542Z
paper_authors._id: 6494bbbb185ae221545a0ef6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6494bbbb185ae221545a0ef6/oA6HXlD0KnNfttYf2f_qD.jpeg
paper_authors.fullname: Ziru Chen
paper_authors.type: user
paper_authors.user: ronch99
paper_authors._id: 670541503262801fdbc1f7be
paper_authors.name: Shijie Chen
paper_authors._id: 670541503262801fdbc1f7bf
paper_authors.name: Yuting Ning
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T07:48:45.268Z
paper_authors._id: 65ace92f64c9b93eca5c2bce
paper_authors.avatarUrl: /avatars/9fca9d018ba751a9dba79621bf0c83f1.svg
paper_authors.fullname: Yuting Ning
paper_authors.type: user
paper_authors.user: nnnyt
paper_authors._id: 670541503262801fdbc1f7c0
paper_authors.name: Qianheng Zhang
paper_authors._id: 670541503262801fdbc1f7c1
paper_authors.name: Boshi Wang
paper_authors._id: 670541503262801fdbc1f7c2
paper_authors.name: Botao Yu
paper_authors._id: 670541503262801fdbc1f7c3
paper_authors.name: Yifei Li
paper_authors._id: 670541503262801fdbc1f7c4
paper_authors.name: Zeyi Liao
paper_authors._id: 670541503262801fdbc1f7c5
paper_authors.name: Chen Wei
paper_authors._id: 670541503262801fdbc1f7c6
paper_authors.name: Zitong Lu
paper_authors._id: 670541503262801fdbc1f7c7
paper_authors.name: Vishal Dey
paper_authors._id: 670541503262801fdbc1f7c8
paper_authors.name: Mingyi Xue
paper_authors._id: 670541503262801fdbc1f7c9
paper_authors.name: Frazier N. Baker
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T15:41:02.694Z
paper_authors._id: 6557abbf03e1f0503bfbfe77
paper_authors.avatarUrl: /avatars/477ecbd68f00353ceee77c5f1832ef73.svg
paper_authors.fullname: Frazier Baker
paper_authors.type: user
paper_authors.user: frazierbaker
paper_authors._id: 670541503262801fdbc1f7ca
paper_authors.name: Benjamin Burns
paper_authors._id: 670541503262801fdbc1f7cb
paper_authors.name: Daniel Adu-Ampratwum
paper_authors._id: 670541503262801fdbc1f7cc
paper_authors.name: Xuhui Huang
paper_authors._id: 670541503262801fdbc1f7cd
paper_authors.name: Xia Ning
paper_authors._id: 670541503262801fdbc1f7ce
paper_authors.name: Song Gao
paper_authors._id: 670541503262801fdbc1f7cf
paper_authors.name: Yu Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T19:21:52.229Z
paper_authors._id: 6477a323dbc2a416f8b852b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/grnmnxO1foz6aRRJ_A2NU.jpeg
paper_authors.fullname: Yu Su
paper_authors.type: user
paper_authors.user: ysu-nlp
paper_authors._id: 670541503262801fdbc1f7d0
paper_authors.name: Huan Sun
paper_title: ScienceAgentBench: Toward Rigorous Assessment of Language Agents for
    Data-Driven Scientific Discovery
paper_summary: The advancements of language language models (LLMs) have piqued growing
  interest in developing LLM-based language agents to automate scientific
  discovery end-to-end, which has sparked both excitement and skepticism about
  the true capabilities of such agents. In this work, we argue that for an agent
  to fully automate scientific discovery, it must be able to complete all
  essential tasks in the workflow. Thus, we call for rigorous assessment of
  agents on individual tasks in a scientific workflow before making bold claims
  on end-to-end automation. To this end, we present ScienceAgentBench, a new
  benchmark for evaluating language agents for data-driven scientific discovery.
  To ensure the scientific authenticity and real-world relevance of our
  benchmark, we extract 102 tasks from 44 peer-reviewed publications in four
  disciplines and engage nine subject matter experts to validate them. We unify
  the target output for every task to a self-contained Python program file and
  employ an array of evaluation metrics to examine the generated programs,
  execution results, and costs. Each task goes through multiple rounds of manual
  validation by annotators and subject matter experts to ensure its annotation
  quality and scientific plausibility. We also propose two effective strategies
  to mitigate data contamination concerns. Using our benchmark, we evaluate five
  open-weight and proprietary LLMs, each with three frameworks: direct prompting,
  OpenHands, and self-debug. Given three attempts for each task, the
  best-performing agent can only solve 32.4% of the tasks independently and 34.3%
  with expert-provided knowledge. These results underscore the limited capacities
  of current language agents in generating code for data-driven discovery, let
  alone end-to-end automation for scientific research.
paper_ai_keywords: language language models (LLMs) language agents scientific discovery scientific workflow ScienceAgentBench peer-reviewed publications subject matter experts self-contained Python program file data-driven scientific discovery direct prompting OpenHands self-debug code generation