Agent-SafetyBench: Evaluating the Safety of LLM Agents
As large language models (LLMs) are increasingly deployed as agents, their
integration into interactive environments and tool use introduce new safety
challenges beyond those associated with the models themselves. However, the
absence of comprehensive benchmarks for evaluating agent safety presents a
significant barrier to effective assessment and further improvement. In this
paper, we introduce Agent-SafetyBench, a comprehensive benchmark designed to
evaluate the safety of LLM agents. Agent-SafetyBench encompasses 349
interaction environments and 2,000 test cases, evaluating 8 categories of
safety risks and covering 10 common failure modes frequently encountered in
unsafe interactions. Our evaluation of 16 popular LLM agents reveals a
concerning result: none of the agents achieves a safety score above 60%. This
highlights significant safety challenges in LLM agents and underscores the
considerable need for improvement. Through quantitative analysis, we identify
critical failure modes and summarize two fundamental safety detects in current
LLM agents: lack of robustness and lack of risk awareness. Furthermore, our
findings suggest that reliance on defense prompts alone is insufficient to
address these safety issues, emphasizing the need for more advanced and robust
strategies. We release Agent-SafetyBench at
https://github.com/thu-coai/Agent-SafetyBench to facilitate further
research and innovation in agent safety evaluation and improvement.
676a77cf5d76485cb34417b3
Zhexin Zhang
claimed_verified
2024-12-30T19:34:43.539Z
61b58aa0d65058ce70beb98c
/avatars/aefd9271b891abc6dd2ded1a30eebca4.svg
Zhexin Zhang
user
nonstopfor
676a77cf5d76485cb34417b4
Shiyao Cui
676a77cf5d76485cb34417b5
Yida Lu
676a77cf5d76485cb34417b6
Jingzhuo Zhou
676a77cf5d76485cb34417b7
Junxiao Yang
claimed_verified
2025-05-22T07:19:18.738Z
65d859a3661492b25c46a117
/avatars/06a547a00b472f512a25eef4ddd047bf.svg
Junxiao Yang
user
yangjunxiao2021
676a77cf5d76485cb34417b8
Hongning Wang
676a77cf5d76485cb34417b9
Minlie Huang
Agent-SafetyBench: Evaluating the Safety of LLM Agents
As large language models (LLMs) are increasingly deployed as agents, their
integration into interactive environments and tool use introduce new safety
challenges beyond those associated with the models themselves. However, the
absence of comprehensive benchmarks for evaluating agent safety presents a
significant barrier to effective assessment and further improvement. In this
paper, we introduce Agent-SafetyBench, a comprehensive benchmark designed to
evaluate the safety of LLM agents. Agent-SafetyBench encompasses 349
interaction environments and 2,000 test cases, evaluating 8 categories of
safety risks and covering 10 common failure modes frequently encountered in
unsafe interactions. Our evaluation of 16 popular LLM agents reveals a
concerning result: none of the agents achieves a safety score above 60%. This
highlights significant safety challenges in LLM agents and underscores the
considerable need for improvement. Through quantitative analysis, we identify
critical failure modes and summarize two fundamental safety detects in current
LLM agents: lack of robustness and lack of risk awareness. Furthermore, our
findings suggest that reliance on defense prompts alone is insufficient to
address these safety issues, emphasizing the need for more advanced and robust
strategies. We release Agent-SafetyBench at
https://github.com/thu-coai/Agent-SafetyBench to facilitate further
research and innovation in agent safety evaluation and improvement.
Agent-SafetyBench evaluates LLM agents in interactive environments, revealing significant safety challenges and emphasizing the need for advanced strategies to improve agent reliability.
large language models LLM agents interactive environments safety benchmarks safety risks failure modes safety score robustness risk awareness defense prompts