title: Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety
    Mechanisms Tend to Be Anchored in The Template Region
summary: The safety alignment of large language models (LLMs) remains vulnerable, as
  their initial behavior can be easily jailbroken by even relatively simple
  attacks. Since infilling a fixed template between the input instruction and
  initial model output is a common practice for existing LLMs, we hypothesize
  that this template is a key factor behind their vulnerabilities: LLMs'
  safety-related decision-making overly relies on the aggregated information from
  the template region, which largely influences these models' safety behavior. We
  refer to this issue as template-anchored safety alignment. In this paper, we
  conduct extensive experiments and verify that template-anchored safety
  alignment is widespread across various aligned LLMs. Our mechanistic analyses
  demonstrate how it leads to models' susceptibility when encountering
  inference-time jailbreak attacks. Furthermore, we show that detaching safety
  mechanisms from the template region is promising in mitigating vulnerabilities
  to jailbreak attacks. We encourage future research to develop more robust
  safety alignment techniques that reduce reliance on the template region.
paper_authors._id: 67b6b416b4ad845374143c31
paper_authors.name: Chak Tou Leong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T09:25:12.631Z
paper_authors._id: 631326d6289cf15634c52369
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631326d6289cf15634c52369/lmPWGHLsQ36H556cqcXjT.jpeg
paper_authors.fullname: Cooper Leong
paper_authors.type: user
paper_authors.user: cooperleong00
paper_authors._id: 67b6b416b4ad845374143c32
paper_authors.name: Qingyu Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T16:12:30.407Z
paper_authors._id: 6453cb22908e259483c0a061
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6453cb22908e259483c0a061/hMgdwZUsUbgquGalzPGzV.jpeg
paper_authors.fullname: Qingyu_Yin
paper_authors.type: user
paper_authors.user: MikaStars39
paper_authors._id: 67b6b416b4ad845374143c33
paper_authors.name: Jian Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T15:53:05.099Z
paper_authors._id: 63a4117984a6a25c65bc2fff
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1675936454785-63a4117984a6a25c65bc2fff.jpeg
paper_authors.fullname: Jian Wang
paper_authors.type: user
paper_authors.user: jwanglvy
paper_authors._id: 67b6b416b4ad845374143c34
paper_authors.name: Wenjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T16:12:36.692Z
paper_authors._id: 66a3710a4ee2a4c936315a5a
paper_authors.avatarUrl: /avatars/ef8da8fb1031695d77d34a5d365aa177.svg
paper_authors.fullname: Li
paper_authors.type: user
paper_authors.user: WenjieLi
paper_title: Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety
    Mechanisms Tend to Be Anchored in The Template Region
paper_summary: The safety alignment of large language models (LLMs) remains vulnerable, as
  their initial behavior can be easily jailbroken by even relatively simple
  attacks. Since infilling a fixed template between the input instruction and
  initial model output is a common practice for existing LLMs, we hypothesize
  that this template is a key factor behind their vulnerabilities: LLMs'
  safety-related decision-making overly relies on the aggregated information from
  the template region, which largely influences these models' safety behavior. We
  refer to this issue as template-anchored safety alignment. In this paper, we
  conduct extensive experiments and verify that template-anchored safety
  alignment is widespread across various aligned LLMs. Our mechanistic analyses
  demonstrate how it leads to models' susceptibility when encountering
  inference-time jailbreak attacks. Furthermore, we show that detaching safety
  mechanisms from the template region is promising in mitigating vulnerabilities
  to jailbreak attacks. We encourage future research to develop more robust
  safety alignment techniques that reduce reliance on the template region.
paper_ai_summary: Template-anchored safety alignment in large language models leads to vulnerabilities to jailbreak attacks, and detaching safety mechanisms from templates can mitigate these vulnerabilities.
paper_ai_keywords: large language models LLMs jailbreak attacks template-anchored safety alignment safety alignment infilling inference-time jailbreak attacks