Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety
  Mechanisms Tend to Be Anchored in The Template Region
The safety alignment of large language models (LLMs) remains vulnerable, as
their initial behavior can be easily jailbroken by even relatively simple
attacks. Since infilling a fixed template between the input instruction and
initial model output is a common practice for existing LLMs, we hypothesize
that this template is a key factor behind their vulnerabilities: LLMs'
safety-related decision-making overly relies on the aggregated information from
the template region, which largely influences these models' safety behavior. We
refer to this issue as template-anchored safety alignment. In this paper, we
conduct extensive experiments and verify that template-anchored safety
alignment is widespread across various aligned LLMs. Our mechanistic analyses
demonstrate how it leads to models' susceptibility when encountering
inference-time jailbreak attacks. Furthermore, we show that detaching safety
mechanisms from the template region is promising in mitigating vulnerabilities
to jailbreak attacks. We encourage future research to develop more robust
safety alignment techniques that reduce reliance on the template region.
67b6b416b4ad845374143c31
Chak Tou Leong
claimed_verified
2025-02-24T09:25:12.631Z
631326d6289cf15634c52369
https://cdn-avatars.huggingface.co/v1/production/uploads/631326d6289cf15634c52369/lmPWGHLsQ36H556cqcXjT.jpeg
Cooper Leong
user
cooperleong00
67b6b416b4ad845374143c32
Qingyu Yin
admin_assigned
2025-02-20T16:12:30.407Z
6453cb22908e259483c0a061
https://cdn-avatars.huggingface.co/v1/production/uploads/6453cb22908e259483c0a061/hMgdwZUsUbgquGalzPGzV.jpeg
Qingyu_Yin
user
MikaStars39
67b6b416b4ad845374143c33
Jian Wang
claimed_verified
2025-02-20T15:53:05.099Z
63a4117984a6a25c65bc2fff
https://cdn-avatars.huggingface.co/v1/production/uploads/1675936454785-63a4117984a6a25c65bc2fff.jpeg
Jian Wang
user
jwanglvy
67b6b416b4ad845374143c34
Wenjie Li
admin_assigned
2025-02-20T16:12:36.692Z
66a3710a4ee2a4c936315a5a
/avatars/ef8da8fb1031695d77d34a5d365aa177.svg
Li
user
WenjieLi
Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety
  Mechanisms Tend to Be Anchored in The Template Region
The safety alignment of large language models (LLMs) remains vulnerable, as
their initial behavior can be easily jailbroken by even relatively simple
attacks. Since infilling a fixed template between the input instruction and
initial model output is a common practice for existing LLMs, we hypothesize
that this template is a key factor behind their vulnerabilities: LLMs'
safety-related decision-making overly relies on the aggregated information from
the template region, which largely influences these models' safety behavior. We
refer to this issue as template-anchored safety alignment. In this paper, we
conduct extensive experiments and verify that template-anchored safety
alignment is widespread across various aligned LLMs. Our mechanistic analyses
demonstrate how it leads to models' susceptibility when encountering
inference-time jailbreak attacks. Furthermore, we show that detaching safety
mechanisms from the template region is promising in mitigating vulnerabilities
to jailbreak attacks. We encourage future research to develop more robust
safety alignment techniques that reduce reliance on the template region.
Template-anchored safety alignment in large language models leads to vulnerabilities to jailbreak attacks, and detaching safety mechanisms from templates can mitigate these vulnerabilities.
large language models LLMs jailbreak attacks template-anchored safety alignment safety alignment infilling inference-time jailbreak attacks