title: o3-mini vs DeepSeek-R1: Which One is Safer?
summary: The irruption of DeepSeek-R1 constitutes a turning point for the AI industry
  in general and the LLMs in particular. Its capabilities have demonstrated
  outstanding performance in several tasks, including creative thinking, code
  generation, maths and automated program repair, at apparently lower execution
  cost. However, LLMs must adhere to an important qualitative property, i.e.,
  their alignment with safety and human values. A clear competitor of DeepSeek-R1
  is its American counterpart, OpenAI's o3-mini model, which is expected to set
  high standards in terms of performance, safety and cost. In this paper we
  conduct a systematic assessment of the safety level of both, DeepSeek-R1 (70b
  version) and OpenAI's o3-mini (beta version). To this end, we make use of our
  recently released automated safety testing tool, named ASTRAL. By leveraging
  this tool, we automatically and systematically generate and execute a total of
  1260 unsafe test inputs on both models. After conducting a semi-automated
  assessment of the outcomes provided by both LLMs, the results indicate that
  DeepSeek-R1 is highly unsafe as compared to OpenAI's o3-mini. Based on our
  evaluation, DeepSeek-R1 answered unsafely to 11.98% of the executed prompts
  whereas o3-mini only to 1.19%.
paper_authors._id: 679c7d0ebd893fb2b7159aa3
paper_authors.name: Aitor Arrieta
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-01-31T07:34:38.875Z
paper_authors._id: 657b3a44de028a439ea2ed9d
paper_authors.avatarUrl: /avatars/9f05e8eb6809a0ce1b50cd1fc9b5a044.svg
paper_authors.fullname: Aitor Arrieta
paper_authors.type: user
paper_authors.user: aitorarrieta
paper_authors._id: 679c7d0ebd893fb2b7159aa4
paper_authors.name: Miriam Ugarte
paper_authors._id: 679c7d0ebd893fb2b7159aa5
paper_authors.name: Pablo Valle
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-31T08:35:44.931Z
paper_authors._id: 65001514f322f9156663f096
paper_authors.avatarUrl: /avatars/e8712f60d4e8b7c70ac02c532ad547ef.svg
paper_authors.fullname: Pablo Valle
paper_authors.type: user
paper_authors.user: pablovalle
paper_authors._id: 679c7d0ebd893fb2b7159aa6
paper_authors.name: José Antonio Parejo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-31T08:49:45.440Z
paper_authors._id: 63527de67e4cc3135fd16651
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63527de67e4cc3135fd16651/bkeQlJEwsPs3E4EsvmmLB.jpeg
paper_authors.fullname: José Antonio Parejo Maestre
paper_authors.type: user
paper_authors.user: japarejo
paper_authors._id: 679c7d0ebd893fb2b7159aa7
paper_authors.name: Sergio Segura
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-02-02T16:41:28.081Z
paper_authors._id: 6790d642a1863df579840ae3
paper_authors.avatarUrl: /avatars/a10a6f4af327c1bb67513c56d7f84820.svg
paper_authors.fullname: Sergio Segura
paper_authors.type: user
paper_authors.user: ssegura
paper_title: o3-mini vs DeepSeek-R1: Which One is Safer?
paper_summary: The irruption of DeepSeek-R1 constitutes a turning point for the AI industry
  in general and the LLMs in particular. Its capabilities have demonstrated
  outstanding performance in several tasks, including creative thinking, code
  generation, maths and automated program repair, at apparently lower execution
  cost. However, LLMs must adhere to an important qualitative property, i.e.,
  their alignment with safety and human values. A clear competitor of DeepSeek-R1
  is its American counterpart, OpenAI's o3-mini model, which is expected to set
  high standards in terms of performance, safety and cost. In this paper we
  conduct a systematic assessment of the safety level of both, DeepSeek-R1 (70b
  version) and OpenAI's o3-mini (beta version). To this end, we make use of our
  recently released automated safety testing tool, named ASTRAL. By leveraging
  this tool, we automatically and systematically generate and execute a total of
  1260 unsafe test inputs on both models. After conducting a semi-automated
  assessment of the outcomes provided by both LLMs, the results indicate that
  DeepSeek-R1 is highly unsafe as compared to OpenAI's o3-mini. Based on our
  evaluation, DeepSeek-R1 answered unsafely to 11.98% of the executed prompts
  whereas o3-mini only to 1.19%.
paper_ai_summary: DeepSeek-R1 and OpenAI's o3-mini are assessed for safety using ASTRAL, with OpenAI's model showing significantly higher safety performance.
paper_ai_keywords: LLMs DeepSeek-R1 o3-mini automated safety testing ASTRAL unsafe test inputs safety level assessment automated program repair