title: ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic
    Scientific Workflows
summary: Large Language Models (LLMs) have extended their impact beyond Natural
  Language Processing, substantially fostering the development of
  interdisciplinary research. Recently, various LLM-based agents have been
  developed to assist scientific discovery progress across multiple aspects and
  domains. Among these, computer-using agents, capable of interacting with
  operating systems as humans do, are paving the way to automated scientific
  problem-solving and addressing routines in researchers' workflows. Recognizing
  the transformative potential of these agents, we introduce ScienceBoard, which
  encompasses two complementary contributions: (i) a realistic, multi-domain
  environment featuring dynamic and visually rich scientific workflows with
  integrated professional software, where agents can autonomously interact via
  different interfaces to accelerate complex research tasks and experiments; and
  (ii) a challenging benchmark of 169 high-quality, rigorously validated
  real-world tasks curated by humans, spanning scientific-discovery workflows in
  domains such as biochemistry, astronomy, and geoinformatics. Extensive
  evaluations of agents with state-of-the-art backbones (e.g., GPT-4o, Claude
  3.7, UI-TARS) show that, despite some promising results, they still fall short
  of reliably assisting scientists in complex workflows, achieving only a 15%
  overall success rate. In-depth analysis further provides valuable insights for
  addressing current agent limitations and more effective design principles,
  paving the way to build more capable agents for scientific discovery. Our code,
  environment, and benchmark are at
  https://qiushisun.github.io/ScienceBoard-Home/.
paper_authors._id: 6835e75d649a767a0771f8a4
paper_authors.name: Qiushi Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T08:59:14.344Z
paper_authors._id: 6064a0eeb1703ddba0d458b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1617207525789-noauth.png
paper_authors.fullname: Qiushi
paper_authors.type: user
paper_authors.user: QiushiSun
paper_authors._id: 6835e75d649a767a0771f8a5
paper_authors.name: Zhoumianze Liu
paper_authors._id: 6835e75d649a767a0771f8a6
paper_authors.name: Chang Ma
paper_authors._id: 6835e75d649a767a0771f8a7
paper_authors.name: Zichen Ding
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T08:59:08.025Z
paper_authors._id: 642b9861bb77f8456634b048
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642b9861bb77f8456634b048/ZT-oJrw5BsADC-gZT_i25.jpeg
paper_authors.fullname: Zichen Ding
paper_authors.type: user
paper_authors.user: heroding77
paper_authors._id: 6835e75d649a767a0771f8a8
paper_authors.name: Fangzhi Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T08:59:12.170Z
paper_authors._id: 64e6cf78ecce34cb442dc889
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e6cf78ecce34cb442dc889/qVZFiUEpBpSkmH8SQeinm.jpeg
paper_authors.fullname: Fangzhi Xu
paper_authors.type: user
paper_authors.user: xufangzhi
paper_authors._id: 6835e75d649a767a0771f8a9
paper_authors.name: Zhangyue Yin
paper_authors._id: 6835e75d649a767a0771f8aa
paper_authors.name: Haiteng Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T19:14:29.511Z
paper_authors._id: 64a7c6e223622f7f189bcbe1
paper_authors.avatarUrl: /avatars/4f13a7ed0d2b8d8dfff7dc650e46450a.svg
paper_authors.fullname: haiteng zhao
paper_authors.type: user
paper_authors.user: haitengzhao
paper_authors._id: 6835e75d649a767a0771f8ab
paper_authors.name: Zhenyu Wu
paper_authors._id: 6835e75d649a767a0771f8ac
paper_authors.name: Kanzhi Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T08:59:03.364Z
paper_authors._id: 63340dbbd92c5842ae71d1e9
paper_authors.avatarUrl: /avatars/3a3182996bd41b526dcbfa8687d91963.svg
paper_authors.fullname: Kanzhi Cheng
paper_authors.type: user
paper_authors.user: cckevinn
paper_authors._id: 6835e75d649a767a0771f8ad
paper_authors.name: Zhaoyang Liu
paper_authors._id: 6835e75d649a767a0771f8ae
paper_authors.name: Jianing Wang
paper_authors._id: 6835e75d649a767a0771f8af
paper_authors.name: Qintong Li
paper_authors._id: 6835e75d649a767a0771f8b0
paper_authors.name: Xiangru Tang
paper_authors._id: 6835e75d649a767a0771f8b1
paper_authors.name: Tianbao Xie
paper_authors._id: 6835e75d649a767a0771f8b2
paper_authors.name: Xiachong Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T08:59:05.718Z
paper_authors._id: 6776ae0c91b4c75dac91249c
paper_authors.avatarUrl: /avatars/a43139f65ad7086426d9757a1bcb7080.svg
paper_authors.fullname: Oran Feng
paper_authors.type: user
paper_authors.user: xiachongfeng
paper_authors._id: 6835e75d649a767a0771f8b3
paper_authors.name: Xiang Li
paper_authors._id: 6835e75d649a767a0771f8b4
paper_authors.name: Ben Kao
paper_authors._id: 6835e75d649a767a0771f8b5
paper_authors.name: Wenhai Wang
paper_authors._id: 6835e75d649a767a0771f8b6
paper_authors.name: Biqing Qi
paper_authors._id: 6835e75d649a767a0771f8b7
paper_authors.name: Lingpeng Kong
paper_authors._id: 6835e75d649a767a0771f8b8
paper_authors.name: Zhiyong Wu
paper_title: ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic
    Scientific Workflows
paper_summary: Large Language Models (LLMs) have extended their impact beyond Natural
  Language Processing, substantially fostering the development of
  interdisciplinary research. Recently, various LLM-based agents have been
  developed to assist scientific discovery progress across multiple aspects and
  domains. Among these, computer-using agents, capable of interacting with
  operating systems as humans do, are paving the way to automated scientific
  problem-solving and addressing routines in researchers' workflows. Recognizing
  the transformative potential of these agents, we introduce ScienceBoard, which
  encompasses two complementary contributions: (i) a realistic, multi-domain
  environment featuring dynamic and visually rich scientific workflows with
  integrated professional software, where agents can autonomously interact via
  different interfaces to accelerate complex research tasks and experiments; and
  (ii) a challenging benchmark of 169 high-quality, rigorously validated
  real-world tasks curated by humans, spanning scientific-discovery workflows in
  domains such as biochemistry, astronomy, and geoinformatics. Extensive
  evaluations of agents with state-of-the-art backbones (e.g., GPT-4o, Claude
  3.7, UI-TARS) show that, despite some promising results, they still fall short
  of reliably assisting scientists in complex workflows, achieving only a 15%
  overall success rate. In-depth analysis further provides valuable insights for
  addressing current agent limitations and more effective design principles,
  paving the way to build more capable agents for scientific discovery. Our code,
  environment, and benchmark are at
  https://qiushisun.github.io/ScienceBoard-Home/.
paper_ai_summary: ScienceBoard provides a realistic scientific workflow environment and benchmark to evaluate the performance of LLM-based agents, demonstrating their current limitations in complex scientific tasks.
paper_ai_keywords: Large Language Models (LLMs) computer-using agents scientific discovery dynamic workflows integrated professional software autonomous interaction ScienceBoard benchmark evaluation state-of-the-art backbones GPT-4o Claude 3.7 UI-TARS biochemistry astronomy geoinformatics design principles