title: Agent-SafetyBench: Evaluating the Safety of LLM Agents
summary: As large language models (LLMs) are increasingly deployed as agents, their
  integration into interactive environments and tool use introduce new safety
  challenges beyond those associated with the models themselves. However, the
  absence of comprehensive benchmarks for evaluating agent safety presents a
  significant barrier to effective assessment and further improvement. In this
  paper, we introduce Agent-SafetyBench, a comprehensive benchmark designed to
  evaluate the safety of LLM agents. Agent-SafetyBench encompasses 349
  interaction environments and 2,000 test cases, evaluating 8 categories of
  safety risks and covering 10 common failure modes frequently encountered in
  unsafe interactions. Our evaluation of 16 popular LLM agents reveals a
  concerning result: none of the agents achieves a safety score above 60%. This
  highlights significant safety challenges in LLM agents and underscores the
  considerable need for improvement. Through quantitative analysis, we identify
  critical failure modes and summarize two fundamental safety detects in current
  LLM agents: lack of robustness and lack of risk awareness. Furthermore, our
  findings suggest that reliance on defense prompts alone is insufficient to
  address these safety issues, emphasizing the need for more advanced and robust
  strategies. We release Agent-SafetyBench at
  https://github.com/thu-coai/Agent-SafetyBench to facilitate further
  research and innovation in agent safety evaluation and improvement.
paper_authors._id: 676a77cf5d76485cb34417b3
paper_authors.name: Zhexin Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:34:43.539Z
paper_authors._id: 61b58aa0d65058ce70beb98c
paper_authors.avatarUrl: /avatars/aefd9271b891abc6dd2ded1a30eebca4.svg
paper_authors.fullname: Zhexin Zhang
paper_authors.type: user
paper_authors.user: nonstopfor
paper_authors._id: 676a77cf5d76485cb34417b4
paper_authors.name: Shiyao Cui
paper_authors._id: 676a77cf5d76485cb34417b5
paper_authors.name: Yida Lu
paper_authors._id: 676a77cf5d76485cb34417b6
paper_authors.name: Jingzhuo Zhou
paper_authors._id: 676a77cf5d76485cb34417b7
paper_authors.name: Junxiao Yang
paper_authors._id: 676a77cf5d76485cb34417b8
paper_authors.name: Hongning Wang
paper_authors._id: 676a77cf5d76485cb34417b9
paper_authors.name: Minlie Huang
paper_title: Agent-SafetyBench: Evaluating the Safety of LLM Agents
paper_summary: As large language models (LLMs) are increasingly deployed as agents, their
  integration into interactive environments and tool use introduce new safety
  challenges beyond those associated with the models themselves. However, the
  absence of comprehensive benchmarks for evaluating agent safety presents a
  significant barrier to effective assessment and further improvement. In this
  paper, we introduce Agent-SafetyBench, a comprehensive benchmark designed to
  evaluate the safety of LLM agents. Agent-SafetyBench encompasses 349
  interaction environments and 2,000 test cases, evaluating 8 categories of
  safety risks and covering 10 common failure modes frequently encountered in
  unsafe interactions. Our evaluation of 16 popular LLM agents reveals a
  concerning result: none of the agents achieves a safety score above 60%. This
  highlights significant safety challenges in LLM agents and underscores the
  considerable need for improvement. Through quantitative analysis, we identify
  critical failure modes and summarize two fundamental safety detects in current
  LLM agents: lack of robustness and lack of risk awareness. Furthermore, our
  findings suggest that reliance on defense prompts alone is insufficient to
  address these safety issues, emphasizing the need for more advanced and robust
  strategies. We release Agent-SafetyBench at
  https://github.com/thu-coai/Agent-SafetyBench to facilitate further
  research and innovation in agent safety evaluation and improvement.
paper_ai_keywords: Agent-SafetyBench interactive environments tool use safety challenges safety assessment safety risks failure modes safety score robustness risk awareness defense prompts