title: Towards Safety Reasoning in LLMs: AI-agentic Deliberation for
    Policy-embedded CoT Data Creation
summary: Safety reasoning is a recent paradigm where LLMs reason over safety policies
  before generating responses, thereby mitigating limitations in existing safety
  measures such as over-refusal and jailbreak vulnerabilities. However,
  implementing this paradigm is challenging due to the resource-intensive process
  of creating high-quality policy-embedded chain-of-thought (CoT) datasets while
  ensuring reasoning remains accurate and free from hallucinations or policy
  conflicts. To tackle this, we propose AIDSAFE: Agentic Iterative Deliberation
  for Safety Reasoning, a novel data generation recipe that leverages multi-agent
  deliberation to iteratively expand reasoning on safety policies. A data refiner
  stage in AIDSAFE ensures high-quality outputs by eliminating repetitive,
  redundant, and deceptive thoughts. AIDSAFE-generated CoTs provide a strong
  foundation for supervised fine-tuning (SFT)-based safety training.
  Additionally, to address the need of preference data in alignment stages, such
  as DPO training, we introduce a supplemental recipe that uses belief
  augmentation to create distinct selected and rejected CoT samples. Our
  evaluations demonstrate that AIDSAFE-generated CoTs achieve superior policy
  adherence and reasoning quality. Consequently, we show that fine-tuning
  open-source LLMs on these CoTs can significantly improve safety generalization
  and jailbreak robustness while maintaining acceptable utility and over-refusal
  accuracy. AIDSAFE-generated CoT datasets can be found here:
  https://huggingface.co/datasets/AmazonScience/AIDSAFE
paper_authors._id: 6838d672f527444e97b746db
paper_authors.name: Tharindu Kumarage
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:47:34.691Z
paper_authors._id: 64779935950ad3b14ddb2a3b
paper_authors.avatarUrl: /avatars/7e5f6880c3edf1c6cba8f017e2d82239.svg
paper_authors.fullname: Tharindu S Kumarage
paper_authors.type: user
paper_authors.user: TharinduSK
paper_authors._id: 6838d672f527444e97b746dc
paper_authors.name: Ninareh Mehrabi
paper_authors._id: 6838d672f527444e97b746dd
paper_authors.name: Anil Ramakrishna
paper_authors._id: 6838d672f527444e97b746de
paper_authors.name: Xinyan Zhao
paper_authors._id: 6838d672f527444e97b746df
paper_authors.name: Richard Zemel
paper_authors._id: 6838d672f527444e97b746e0
paper_authors.name: Kai-Wei Chang
paper_authors._id: 6838d672f527444e97b746e1
paper_authors.name: Aram Galstyan
paper_authors._id: 6838d672f527444e97b746e2
paper_authors.name: Rahul Gupta
paper_authors._id: 6838d672f527444e97b746e3
paper_authors.name: Charith Peris
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:47:37.344Z
paper_authors._id: 626029fdd47e3dbae32d89e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1651094348689-626029fdd47e3dbae32d89e2.jpeg
paper_authors.fullname: Charith Peris
paper_authors.type: user
paper_authors.user: cperiz
paper_title: Towards Safety Reasoning in LLMs: AI-agentic Deliberation for
    Policy-embedded CoT Data Creation
paper_summary: Safety reasoning is a recent paradigm where LLMs reason over safety policies
  before generating responses, thereby mitigating limitations in existing safety
  measures such as over-refusal and jailbreak vulnerabilities. However,
  implementing this paradigm is challenging due to the resource-intensive process
  of creating high-quality policy-embedded chain-of-thought (CoT) datasets while
  ensuring reasoning remains accurate and free from hallucinations or policy
  conflicts. To tackle this, we propose AIDSAFE: Agentic Iterative Deliberation
  for Safety Reasoning, a novel data generation recipe that leverages multi-agent
  deliberation to iteratively expand reasoning on safety policies. A data refiner
  stage in AIDSAFE ensures high-quality outputs by eliminating repetitive,
  redundant, and deceptive thoughts. AIDSAFE-generated CoTs provide a strong
  foundation for supervised fine-tuning (SFT)-based safety training.
  Additionally, to address the need of preference data in alignment stages, such
  as DPO training, we introduce a supplemental recipe that uses belief
  augmentation to create distinct selected and rejected CoT samples. Our
  evaluations demonstrate that AIDSAFE-generated CoTs achieve superior policy
  adherence and reasoning quality. Consequently, we show that fine-tuning
  open-source LLMs on these CoTs can significantly improve safety generalization
  and jailbreak robustness while maintaining acceptable utility and over-refusal
  accuracy. AIDSAFE-generated CoT datasets can be found here:
  https://huggingface.co/datasets/AmazonScience/AIDSAFE
paper_ai_summary: AIDSAFE uses multi-agent deliberation to create high-quality safety policy datasets, enhancing LLM safety without compromising utility.
paper_ai_keywords: LLMs safety reasoning policy-embedded chain-of-thought CoT data generation multi-agent deliberation data refiner supervised fine-tuning belief augmentation DPO training policy adherence reasoning quality safety generalization jailbreak robustness utility over-refusal accuracy