title: SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning
summary: Large Reasoning Models (LRMs) introduce a new generation paradigm of
  explicitly reasoning before answering, leading to remarkable improvements in
  complex tasks. However, they pose great safety risks against harmful queries
  and adversarial attacks. While recent mainstream safety efforts on LRMs,
  supervised fine-tuning (SFT), improve safety performance, we find that
  SFT-aligned models struggle to generalize to unseen jailbreak prompts. After
  thorough investigation of LRMs' generation, we identify a safety aha moment
  that can activate safety reasoning and lead to a safe response. This aha moment
  typically appears in the `key sentence', which follows models' query
  understanding process and can indicate whether the model will proceed safely.
  Based on these insights, we propose SafeKey, including two complementary
  objectives to better activate the safety aha moment in the key sentence: (1) a
  Dual-Path Safety Head to enhance the safety signal in the model's internal
  representations before the key sentence, and (2) a Query-Mask Modeling
  objective to improve the models' attention on its query understanding, which
  has important safety hints. Experiments across multiple safety benchmarks
  demonstrate that our methods significantly improve safety generalization to a
  wide range of jailbreak attacks and out-of-distribution harmful prompts,
  lowering the average harmfulness rate by 9.6\%, while maintaining general
  abilities. Our analysis reveals how SafeKey enhances safety by reshaping
  internal attention and improving the quality of hidden representations.
paper_authors._id: 683005352b4a4d1ce546568b
paper_authors.name: Kaiwen Zhou
paper_authors._id: 683005352b4a4d1ce546568c
paper_authors.name: Xuandong Zhao
paper_authors._id: 683005352b4a4d1ce546568d
paper_authors.name: Gaowen Liu
paper_authors._id: 683005352b4a4d1ce546568e
paper_authors.name: Jayanth Srinivasa
paper_authors._id: 683005352b4a4d1ce546568f
paper_authors.name: Aosong Feng
paper_authors._id: 683005352b4a4d1ce5465690
paper_authors.name: Dawn Song
paper_authors._id: 683005352b4a4d1ce5465691
paper_authors.name: Xin Eric Wang
paper_title: SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning
paper_summary: Large Reasoning Models (LRMs) introduce a new generation paradigm of
  explicitly reasoning before answering, leading to remarkable improvements in
  complex tasks. However, they pose great safety risks against harmful queries
  and adversarial attacks. While recent mainstream safety efforts on LRMs,
  supervised fine-tuning (SFT), improve safety performance, we find that
  SFT-aligned models struggle to generalize to unseen jailbreak prompts. After
  thorough investigation of LRMs' generation, we identify a safety aha moment
  that can activate safety reasoning and lead to a safe response. This aha moment
  typically appears in the `key sentence', which follows models' query
  understanding process and can indicate whether the model will proceed safely.
  Based on these insights, we propose SafeKey, including two complementary
  objectives to better activate the safety aha moment in the key sentence: (1) a
  Dual-Path Safety Head to enhance the safety signal in the model's internal
  representations before the key sentence, and (2) a Query-Mask Modeling
  objective to improve the models' attention on its query understanding, which
  has important safety hints. Experiments across multiple safety benchmarks
  demonstrate that our methods significantly improve safety generalization to a
  wide range of jailbreak attacks and out-of-distribution harmful prompts,
  lowering the average harmfulness rate by 9.6\%, while maintaining general
  abilities. Our analysis reveals how SafeKey enhances safety by reshaping
  internal attention and improving the quality of hidden representations.
paper_ai_summary: SafeKey enhances the safety of large reasoning models by focusing on activating a safety aha moment in the key sentence through dual-path safety head and query-mask modeling, thereby improving generalization to harmful prompts.
paper_ai_keywords: Large Reasoning Models LRMs explicit reasoning safety risks adversarial attacks supervised fine-tuning SFT safety aha moment key sentence query understanding process Dual-Path Safety Head Query-Mask Modeling safety generalization jailbreak attacks out-of-distribution harmful prompts average harmfulness rate hidden representations