o3-mini vs DeepSeek-R1: Which One is Safer?
The irruption of DeepSeek-R1 constitutes a turning point for the AI industry
in general and the LLMs in particular. Its capabilities have demonstrated
outstanding performance in several tasks, including creative thinking, code
generation, maths and automated program repair, at apparently lower execution
cost. However, LLMs must adhere to an important qualitative property, i.e.,
their alignment with safety and human values. A clear competitor of DeepSeek-R1
is its American counterpart, OpenAI's o3-mini model, which is expected to set
high standards in terms of performance, safety and cost. In this paper we
conduct a systematic assessment of the safety level of both, DeepSeek-R1 (70b
version) and OpenAI's o3-mini (beta version). To this end, we make use of our
recently released automated safety testing tool, named ASTRAL. By leveraging
this tool, we automatically and systematically generate and execute a total of
1260 unsafe test inputs on both models. After conducting a semi-automated
assessment of the outcomes provided by both LLMs, the results indicate that
DeepSeek-R1 is highly unsafe as compared to OpenAI's o3-mini. Based on our
evaluation, DeepSeek-R1 answered unsafely to 11.98% of the executed prompts
whereas o3-mini only to 1.19%.
679c7d0ebd893fb2b7159aa3
Aitor Arrieta
extracted_pending
2025-01-31T07:34:38.875Z
657b3a44de028a439ea2ed9d
/avatars/9f05e8eb6809a0ce1b50cd1fc9b5a044.svg
Aitor Arrieta
user
aitorarrieta
679c7d0ebd893fb2b7159aa4
Miriam Ugarte
679c7d0ebd893fb2b7159aa5
Pablo Valle
claimed_verified
2025-01-31T08:35:44.931Z
65001514f322f9156663f096
/avatars/e8712f60d4e8b7c70ac02c532ad547ef.svg
Pablo Valle
user
pablovalle
679c7d0ebd893fb2b7159aa6
José Antonio Parejo
admin_assigned
2025-01-31T08:49:45.440Z
63527de67e4cc3135fd16651
https://cdn-avatars.huggingface.co/v1/production/uploads/63527de67e4cc3135fd16651/bkeQlJEwsPs3E4EsvmmLB.jpeg
José Antonio Parejo Maestre
user
japarejo
679c7d0ebd893fb2b7159aa7
Sergio Segura
extracted_confirmed
2025-02-02T16:41:28.081Z
6790d642a1863df579840ae3
/avatars/a10a6f4af327c1bb67513c56d7f84820.svg
Sergio Segura
user
ssegura
o3-mini vs DeepSeek-R1: Which One is Safer?
The irruption of DeepSeek-R1 constitutes a turning point for the AI industry
in general and the LLMs in particular. Its capabilities have demonstrated
outstanding performance in several tasks, including creative thinking, code
generation, maths and automated program repair, at apparently lower execution
cost. However, LLMs must adhere to an important qualitative property, i.e.,
their alignment with safety and human values. A clear competitor of DeepSeek-R1
is its American counterpart, OpenAI's o3-mini model, which is expected to set
high standards in terms of performance, safety and cost. In this paper we
conduct a systematic assessment of the safety level of both, DeepSeek-R1 (70b
version) and OpenAI's o3-mini (beta version). To this end, we make use of our
recently released automated safety testing tool, named ASTRAL. By leveraging
this tool, we automatically and systematically generate and execute a total of
1260 unsafe test inputs on both models. After conducting a semi-automated
assessment of the outcomes provided by both LLMs, the results indicate that
DeepSeek-R1 is highly unsafe as compared to OpenAI's o3-mini. Based on our
evaluation, DeepSeek-R1 answered unsafely to 11.98% of the executed prompts
whereas o3-mini only to 1.19%.
DeepSeek-R1 and OpenAI's o3-mini are assessed for safety using ASTRAL, with OpenAI's model showing significantly higher safety performance.
LLMs DeepSeek-R1 o3-mini automated safety testing ASTRAL unsafe test inputs safety level assessment automated program repair