Early External Safety Testing of OpenAI's o3-mini: Insights from the
  Pre-Deployment Evaluation
Large Language Models (LLMs) have become an integral part of our daily lives.
However, they impose certain risks, including those that can harm individuals'
privacy, perpetuate biases and spread misinformation. These risks highlight the
need for robust safety mechanisms, ethical guidelines, and thorough testing to
ensure their responsible deployment. Safety of LLMs is a key property that
needs to be thoroughly tested prior the model to be deployed and accessible to
the general users. This paper reports the external safety testing experience
conducted by researchers from Mondragon University and University of Seville on
OpenAI's new o3-mini LLM as part of OpenAI's early access for safety testing
program. In particular, we apply our tool, ASTRAL, to automatically and
systematically generate up to date unsafe test inputs (i.e., prompts) that
helps us test and assess different safety categories of LLMs. We automatically
generate and execute a total of 10,080 unsafe test input on a early o3-mini
beta version. After manually verifying the test cases classified as unsafe by
ASTRAL, we identify a total of 87 actual instances of unsafe LLM behavior. We
highlight key insights and findings uncovered during the pre-deployment
external testing phase of OpenAI's latest LLM.
679ae5eab898ac90bf4480b6
Aitor Arrieta
extracted_confirmed
2025-01-30T08:45:20.561Z
657b3a44de028a439ea2ed9d
/avatars/9f05e8eb6809a0ce1b50cd1fc9b5a044.svg
Aitor Arrieta
user
aitorarrieta
679ae5eab898ac90bf4480b7
Miriam Ugarte
679ae5eab898ac90bf4480b8
Pablo Valle
admin_assigned
2025-01-30T09:39:30.629Z
65001514f322f9156663f096
/avatars/e8712f60d4e8b7c70ac02c532ad547ef.svg
Pablo Valle
user
pablovalle
679ae5eab898ac90bf4480b9
José Antonio Parejo
admin_assigned
2025-01-30T09:39:06.958Z
63527de67e4cc3135fd16651
https://cdn-avatars.huggingface.co/v1/production/uploads/63527de67e4cc3135fd16651/bkeQlJEwsPs3E4EsvmmLB.jpeg
José Antonio Parejo Maestre
user
japarejo
679ae5eab898ac90bf4480ba
Sergio Segura
extracted_confirmed
2025-02-02T16:41:27.270Z
6790d642a1863df579840ae3
/avatars/a10a6f4af327c1bb67513c56d7f84820.svg
Sergio Segura
user
ssegura
Early External Safety Testing of OpenAI's o3-mini: Insights from the
  Pre-Deployment Evaluation
Large Language Models (LLMs) have become an integral part of our daily lives.
However, they impose certain risks, including those that can harm individuals'
privacy, perpetuate biases and spread misinformation. These risks highlight the
need for robust safety mechanisms, ethical guidelines, and thorough testing to
ensure their responsible deployment. Safety of LLMs is a key property that
needs to be thoroughly tested prior the model to be deployed and accessible to
the general users. This paper reports the external safety testing experience
conducted by researchers from Mondragon University and University of Seville on
OpenAI's new o3-mini LLM as part of OpenAI's early access for safety testing
program. In particular, we apply our tool, ASTRAL, to automatically and
systematically generate up to date unsafe test inputs (i.e., prompts) that
helps us test and assess different safety categories of LLMs. We automatically
generate and execute a total of 10,080 unsafe test input on a early o3-mini
beta version. After manually verifying the test cases classified as unsafe by
ASTRAL, we identify a total of 87 actual instances of unsafe LLM behavior. We
highlight key insights and findings uncovered during the pre-deployment
external testing phase of OpenAI's latest LLM.
Researchers used ASTRAL to generate and test unsafe inputs on OpenAI's o3-mini LLM, identifying 87 instances of unsafe behavior before deployment.
Large Language Models LLMs ASTRAL safety testing unsafe prompts safety categories unsafe behavior