SoK: Evaluating Jailbreak Guardrails for Large Language Models
Large Language Models (LLMs) have achieved remarkable progress, but their
deployment has exposed critical vulnerabilities, particularly to jailbreak
attacks that circumvent safety mechanisms. Guardrails--external defense
mechanisms that monitor and control LLM interaction--have emerged as a
promising solution. However, the current landscape of LLM guardrails is
fragmented, lacking a unified taxonomy and comprehensive evaluation framework.
In this Systematization of Knowledge (SoK) paper, we present the first holistic
analysis of jailbreak guardrails for LLMs. We propose a novel,
multi-dimensional taxonomy that categorizes guardrails along six key
dimensions, and introduce a Security-Efficiency-Utility evaluation framework to
assess their practical effectiveness. Through extensive analysis and
experiments, we identify the strengths and limitations of existing guardrail
approaches, explore their universality across attack types, and provide
insights into optimizing defense combinations. Our work offers a structured
foundation for future research and development, aiming to guide the principled
advancement and deployment of robust LLM guardrails. The code is available at
https://github.com/xunguangwang/SoK4JailbreakGuardrails.
685a0fb40e4ad7e219758522
Xunguang Wang
admin_assigned
2025-06-24T17:43:21.005Z
6601853162471e0981261241
/avatars/ccd1c5ce9d2f6fe7c2aff80fd9c39270.svg
XunguangWang
user
xunguangwang
685a0fb40e4ad7e219758523
Zhenlan Ji
685a0fb40e4ad7e219758524
Wenxuan Wang
685a0fb40e4ad7e219758525
Zongjie Li
685a0fb40e4ad7e219758526
Daoyuan Wu
685a0fb40e4ad7e219758527
Shuai Wang
SoK: Evaluating Jailbreak Guardrails for Large Language Models
Large Language Models (LLMs) have achieved remarkable progress, but their
deployment has exposed critical vulnerabilities, particularly to jailbreak
attacks that circumvent safety mechanisms. Guardrails--external defense
mechanisms that monitor and control LLM interaction--have emerged as a
promising solution. However, the current landscape of LLM guardrails is
fragmented, lacking a unified taxonomy and comprehensive evaluation framework.
In this Systematization of Knowledge (SoK) paper, we present the first holistic
analysis of jailbreak guardrails for LLMs. We propose a novel,
multi-dimensional taxonomy that categorizes guardrails along six key
dimensions, and introduce a Security-Efficiency-Utility evaluation framework to
assess their practical effectiveness. Through extensive analysis and
experiments, we identify the strengths and limitations of existing guardrail
approaches, explore their universality across attack types, and provide
insights into optimizing defense combinations. Our work offers a structured
foundation for future research and development, aiming to guide the principled
advancement and deployment of robust LLM guardrails. The code is available at
https://github.com/xunguangwang/SoK4JailbreakGuardrails.
A systematic analysis and evaluation framework for jailbreak guardrails in Large Language Models is presented, categorizing and assessing their effectiveness and optimization potential.
Large Language Models LLMs jailbreak attacks guardrails Security-Efficiency-Utility framework multi-dimensional taxonomy