ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic
  Scientific Workflows
Large Language Models (LLMs) have extended their impact beyond Natural
Language Processing, substantially fostering the development of
interdisciplinary research. Recently, various LLM-based agents have been
developed to assist scientific discovery progress across multiple aspects and
domains. Among these, computer-using agents, capable of interacting with
operating systems as humans do, are paving the way to automated scientific
problem-solving and addressing routines in researchers' workflows. Recognizing
the transformative potential of these agents, we introduce ScienceBoard, which
encompasses two complementary contributions: (i) a realistic, multi-domain
environment featuring dynamic and visually rich scientific workflows with
integrated professional software, where agents can autonomously interact via
different interfaces to accelerate complex research tasks and experiments; and
(ii) a challenging benchmark of 169 high-quality, rigorously validated
real-world tasks curated by humans, spanning scientific-discovery workflows in
domains such as biochemistry, astronomy, and geoinformatics. Extensive
evaluations of agents with state-of-the-art backbones (e.g., GPT-4o, Claude
3.7, UI-TARS) show that, despite some promising results, they still fall short
of reliably assisting scientists in complex workflows, achieving only a 15%
overall success rate. In-depth analysis further provides valuable insights for
addressing current agent limitations and more effective design principles,
paving the way to build more capable agents for scientific discovery. Our code,
environment, and benchmark are at
https://qiushisun.github.io/ScienceBoard-Home/.
6835e75d649a767a0771f8a4
Qiushi Sun
claimed_verified
2025-05-28T08:59:14.344Z
6064a0eeb1703ddba0d458b9
https://cdn-avatars.huggingface.co/v1/production/uploads/1617207525789-noauth.png
Qiushi
user
QiushiSun
6835e75d649a767a0771f8a5
Zhoumianze Liu
6835e75d649a767a0771f8a6
Chang Ma
6835e75d649a767a0771f8a7
Zichen Ding
claimed_verified
2025-05-28T08:59:08.025Z
642b9861bb77f8456634b048
https://cdn-avatars.huggingface.co/v1/production/uploads/642b9861bb77f8456634b048/ZT-oJrw5BsADC-gZT_i25.jpeg
Zichen Ding
user
heroding77
6835e75d649a767a0771f8a8
Fangzhi Xu
claimed_verified
2025-05-28T08:59:12.170Z
64e6cf78ecce34cb442dc889
https://cdn-avatars.huggingface.co/v1/production/uploads/64e6cf78ecce34cb442dc889/qVZFiUEpBpSkmH8SQeinm.jpeg
Fangzhi Xu
user
xufangzhi
6835e75d649a767a0771f8a9
Zhangyue Yin
6835e75d649a767a0771f8aa
Haiteng Zhao
claimed_verified
2025-05-28T19:14:29.511Z
64a7c6e223622f7f189bcbe1
/avatars/4f13a7ed0d2b8d8dfff7dc650e46450a.svg
haiteng zhao
user
haitengzhao
6835e75d649a767a0771f8ab
Zhenyu Wu
6835e75d649a767a0771f8ac
Kanzhi Cheng
claimed_verified
2025-05-28T08:59:03.364Z
63340dbbd92c5842ae71d1e9
/avatars/3a3182996bd41b526dcbfa8687d91963.svg
Kanzhi Cheng
user
cckevinn
6835e75d649a767a0771f8ad
Zhaoyang Liu
6835e75d649a767a0771f8ae
Jianing Wang
6835e75d649a767a0771f8af
Qintong Li
6835e75d649a767a0771f8b0
Xiangru Tang
6835e75d649a767a0771f8b1
Tianbao Xie
6835e75d649a767a0771f8b2
Xiachong Feng
claimed_verified
2025-05-28T08:59:05.718Z
6776ae0c91b4c75dac91249c
/avatars/a43139f65ad7086426d9757a1bcb7080.svg
Oran Feng
user
xiachongfeng
6835e75d649a767a0771f8b3
Xiang Li
6835e75d649a767a0771f8b4
Ben Kao
6835e75d649a767a0771f8b5
Wenhai Wang
6835e75d649a767a0771f8b6
Biqing Qi
6835e75d649a767a0771f8b7
Lingpeng Kong
6835e75d649a767a0771f8b8
Zhiyong Wu
ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic
  Scientific Workflows
Large Language Models (LLMs) have extended their impact beyond Natural
Language Processing, substantially fostering the development of
interdisciplinary research. Recently, various LLM-based agents have been
developed to assist scientific discovery progress across multiple aspects and
domains. Among these, computer-using agents, capable of interacting with
operating systems as humans do, are paving the way to automated scientific
problem-solving and addressing routines in researchers' workflows. Recognizing
the transformative potential of these agents, we introduce ScienceBoard, which
encompasses two complementary contributions: (i) a realistic, multi-domain
environment featuring dynamic and visually rich scientific workflows with
integrated professional software, where agents can autonomously interact via
different interfaces to accelerate complex research tasks and experiments; and
(ii) a challenging benchmark of 169 high-quality, rigorously validated
real-world tasks curated by humans, spanning scientific-discovery workflows in
domains such as biochemistry, astronomy, and geoinformatics. Extensive
evaluations of agents with state-of-the-art backbones (e.g., GPT-4o, Claude
3.7, UI-TARS) show that, despite some promising results, they still fall short
of reliably assisting scientists in complex workflows, achieving only a 15%
overall success rate. In-depth analysis further provides valuable insights for
addressing current agent limitations and more effective design principles,
paving the way to build more capable agents for scientific discovery. Our code,
environment, and benchmark are at
https://qiushisun.github.io/ScienceBoard-Home/.
ScienceBoard provides a realistic scientific workflow environment and benchmark to evaluate the performance of LLM-based agents, demonstrating their current limitations in complex scientific tasks.
Large Language Models (LLMs) computer-using agents scientific discovery dynamic workflows integrated professional software autonomous interaction ScienceBoard benchmark evaluation state-of-the-art backbones GPT-4o Claude 3.7 UI-TARS biochemistry astronomy geoinformatics design principles