title: Early External Safety Testing of OpenAI's o3-mini: Insights from the
    Pre-Deployment Evaluation
summary: Large Language Models (LLMs) have become an integral part of our daily lives.
  However, they impose certain risks, including those that can harm individuals'
  privacy, perpetuate biases and spread misinformation. These risks highlight the
  need for robust safety mechanisms, ethical guidelines, and thorough testing to
  ensure their responsible deployment. Safety of LLMs is a key property that
  needs to be thoroughly tested prior the model to be deployed and accessible to
  the general users. This paper reports the external safety testing experience
  conducted by researchers from Mondragon University and University of Seville on
  OpenAI's new o3-mini LLM as part of OpenAI's early access for safety testing
  program. In particular, we apply our tool, ASTRAL, to automatically and
  systematically generate up to date unsafe test inputs (i.e., prompts) that
  helps us test and assess different safety categories of LLMs. We automatically
  generate and execute a total of 10,080 unsafe test input on a early o3-mini
  beta version. After manually verifying the test cases classified as unsafe by
  ASTRAL, we identify a total of 87 actual instances of unsafe LLM behavior. We
  highlight key insights and findings uncovered during the pre-deployment
  external testing phase of OpenAI's latest LLM.
paper_authors._id: 679ae5eab898ac90bf4480b6
paper_authors.name: Aitor Arrieta
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-01-30T08:45:20.561Z
paper_authors._id: 657b3a44de028a439ea2ed9d
paper_authors.avatarUrl: /avatars/9f05e8eb6809a0ce1b50cd1fc9b5a044.svg
paper_authors.fullname: Aitor Arrieta
paper_authors.type: user
paper_authors.user: aitorarrieta
paper_authors._id: 679ae5eab898ac90bf4480b7
paper_authors.name: Miriam Ugarte
paper_authors._id: 679ae5eab898ac90bf4480b8
paper_authors.name: Pablo Valle
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-30T09:39:30.629Z
paper_authors._id: 65001514f322f9156663f096
paper_authors.avatarUrl: /avatars/e8712f60d4e8b7c70ac02c532ad547ef.svg
paper_authors.fullname: Pablo Valle
paper_authors.type: user
paper_authors.user: pablovalle
paper_authors._id: 679ae5eab898ac90bf4480b9
paper_authors.name: José Antonio Parejo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-30T09:39:06.958Z
paper_authors._id: 63527de67e4cc3135fd16651
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63527de67e4cc3135fd16651/bkeQlJEwsPs3E4EsvmmLB.jpeg
paper_authors.fullname: José Antonio Parejo Maestre
paper_authors.type: user
paper_authors.user: japarejo
paper_authors._id: 679ae5eab898ac90bf4480ba
paper_authors.name: Sergio Segura
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-02-02T16:41:27.270Z
paper_authors._id: 6790d642a1863df579840ae3
paper_authors.avatarUrl: /avatars/a10a6f4af327c1bb67513c56d7f84820.svg
paper_authors.fullname: Sergio Segura
paper_authors.type: user
paper_authors.user: ssegura
paper_title: Early External Safety Testing of OpenAI's o3-mini: Insights from the
    Pre-Deployment Evaluation
paper_summary: Large Language Models (LLMs) have become an integral part of our daily lives.
  However, they impose certain risks, including those that can harm individuals'
  privacy, perpetuate biases and spread misinformation. These risks highlight the
  need for robust safety mechanisms, ethical guidelines, and thorough testing to
  ensure their responsible deployment. Safety of LLMs is a key property that
  needs to be thoroughly tested prior the model to be deployed and accessible to
  the general users. This paper reports the external safety testing experience
  conducted by researchers from Mondragon University and University of Seville on
  OpenAI's new o3-mini LLM as part of OpenAI's early access for safety testing
  program. In particular, we apply our tool, ASTRAL, to automatically and
  systematically generate up to date unsafe test inputs (i.e., prompts) that
  helps us test and assess different safety categories of LLMs. We automatically
  generate and execute a total of 10,080 unsafe test input on a early o3-mini
  beta version. After manually verifying the test cases classified as unsafe by
  ASTRAL, we identify a total of 87 actual instances of unsafe LLM behavior. We
  highlight key insights and findings uncovered during the pre-deployment
  external testing phase of OpenAI's latest LLM.