title: SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents
summary: Recent advancements in large language model (LLM) agents have significantly
  accelerated scientific discovery automation, yet concurrently raised critical
  ethical and safety concerns. To systematically address these challenges, we
  introduce SafeScientist, an innovative AI scientist framework
  explicitly designed to enhance safety and ethical responsibility in AI-driven
  scientific exploration. SafeScientist proactively refuses ethically
  inappropriate or high-risk tasks and rigorously emphasizes safety throughout
  the research process. To achieve comprehensive safety oversight, we integrate
  multiple defensive mechanisms, including prompt monitoring, agent-collaboration
  monitoring, tool-use monitoring, and an ethical reviewer component.
  Complementing SafeScientist, we propose SciSafetyBench, a novel
  benchmark specifically designed to evaluate AI safety in scientific contexts,
  comprising 240 high-risk scientific tasks across 6 domains, alongside 30
  specially designed scientific tools and 120 tool-related risk tasks. Extensive
  experiments demonstrate that SafeScientist significantly improves safety
  performance by 35\% compared to traditional AI scientist frameworks, without
  compromising scientific output quality. Additionally, we rigorously validate
  the robustness of our safety pipeline against diverse adversarial attack
  methods, further confirming the effectiveness of our integrated approach. The
  code and data will be available at https://github.com/ulab-uiuc/SafeScientist.
  red{Warning: this paper contains example data that may be offensive
  or harmful.}
paper_authors._id: 68390eebc260f5fa36eaa27c
paper_authors.name: Kunlun Zhu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-30T01:50:37.811Z
paper_authors._id: 66554507e6ea63012f35824c
paper_authors.avatarUrl: /avatars/b82de75bd60890e7bb524fc3754b131c.svg
paper_authors.fullname: Kunlun_Zhu
paper_authors.type: user
paper_authors.user: Leozkl
paper_authors._id: 68390eebc260f5fa36eaa27d
paper_authors.name: Jiaxun Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:47:22.497Z
paper_authors._id: 67fab82c6c81d1828aeb86ac
paper_authors.avatarUrl: /avatars/e5f87f8084bc1bb3345887807dd9e991.svg
paper_authors.fullname: Jiaxun Zhang
paper_authors.type: user
paper_authors.user: JiaxunZ
paper_authors._id: 68390eebc260f5fa36eaa27e
paper_authors.name: Ziheng Qi
paper_authors._id: 68390eebc260f5fa36eaa27f
paper_authors.name: Nuoxing Shang
paper_authors._id: 68390eebc260f5fa36eaa280
paper_authors.name: Zijia Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:54:08.570Z
paper_authors._id: 65d188a4aa309d842e438ef1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65d188a4aa309d842e438ef1/t5zmBIytp-xnMoHUfYhEa.jpeg
paper_authors.fullname: Zijia Liu
paper_authors.type: user
paper_authors.user: m-serious
paper_authors._id: 68390eebc260f5fa36eaa281
paper_authors.name: Peixuan Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:54:06.300Z
paper_authors._id: 638d601b5e14c2f38678fb3a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638d601b5e14c2f38678fb3a/Elu-TTd97lGy7YL7eKRuZ.jpeg
paper_authors.fullname: 韩沛煊
paper_authors.type: user
paper_authors.user: HakHan
paper_authors._id: 68390eebc260f5fa36eaa282
paper_authors.name: Yue Su
paper_authors._id: 68390eebc260f5fa36eaa283
paper_authors.name: Haofei Yu
paper_authors._id: 68390eebc260f5fa36eaa284
paper_authors.name: Jiaxuan You
paper_title: SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents
paper_summary: Recent advancements in large language model (LLM) agents have significantly
  accelerated scientific discovery automation, yet concurrently raised critical
  ethical and safety concerns. To systematically address these challenges, we
  introduce SafeScientist, an innovative AI scientist framework
  explicitly designed to enhance safety and ethical responsibility in AI-driven
  scientific exploration. SafeScientist proactively refuses ethically
  inappropriate or high-risk tasks and rigorously emphasizes safety throughout
  the research process. To achieve comprehensive safety oversight, we integrate
  multiple defensive mechanisms, including prompt monitoring, agent-collaboration
  monitoring, tool-use monitoring, and an ethical reviewer component.
  Complementing SafeScientist, we propose SciSafetyBench, a novel
  benchmark specifically designed to evaluate AI safety in scientific contexts,
  comprising 240 high-risk scientific tasks across 6 domains, alongside 30
  specially designed scientific tools and 120 tool-related risk tasks. Extensive
  experiments demonstrate that SafeScientist significantly improves safety
  performance by 35\% compared to traditional AI scientist frameworks, without
  compromising scientific output quality. Additionally, we rigorously validate
  the robustness of our safety pipeline against diverse adversarial attack
  methods, further confirming the effectiveness of our integrated approach. The
  code and data will be available at https://github.com/ulab-uiuc/SafeScientist.
  red{Warning: this paper contains example data that may be offensive
  or harmful.}
paper_ai_summary: SafeScientist is an AI framework that enhances safety in AI-driven scientific research through multiple defensive mechanisms and is validated using the SciSafetyBench benchmark.
paper_ai_keywords: SafeScientist AI scientist framework ethical responsibility prompt monitoring agent-collaboration monitoring tool-use monitoring ethical reviewer component SciSafetyBench benchmark scientific tasks scientific tools safety performance adversarial attack methods