title: Forbidden Science: Dual-Use AI Challenge Benchmark and Scientific
    Refusal Tests
summary: The development of robust safety benchmarks for large language models
  requires open, reproducible datasets that can measure both appropriate refusal
  of harmful content and potential over-restriction of legitimate scientific
  discourse. We present an open-source dataset and testing framework for
  evaluating LLM safety mechanisms across mainly controlled substance queries,
  analyzing four major models' responses to systematically varied prompts. Our
  results reveal distinct safety profiles: Claude-3.5-sonnet demonstrated the
  most conservative approach with 73% refusals and 27% allowances, while Mistral
  attempted to answer 100% of queries. GPT-3.5-turbo showed moderate restriction
  with 10% refusals and 90% allowances, and Grok-2 registered 20% refusals and
  80% allowances. Testing prompt variation strategies revealed decreasing
  response consistency, from 85% with single prompts to 65% with five variations.
  This publicly available benchmark enables systematic evaluation of the critical
  balance between necessary safety restrictions and potential over-censorship of
  legitimate scientific inquiry, while providing a foundation for measuring
  progress in AI safety implementation. Chain-of-thought analysis reveals
  potential vulnerabilities in safety mechanisms, highlighting the complexity of
  implementing robust safeguards without unduly restricting desirable and valid
  scientific discourse.
paper_authors._id: 67ac026e401012b81040ae8b
paper_authors.name: David Noever
paper_authors._id: 67ac026e401012b81040ae8c
paper_authors.name: Forrest McKee
paper_title: Forbidden Science: Dual-Use AI Challenge Benchmark and Scientific
    Refusal Tests
paper_summary: The development of robust safety benchmarks for large language models
  requires open, reproducible datasets that can measure both appropriate refusal
  of harmful content and potential over-restriction of legitimate scientific
  discourse. We present an open-source dataset and testing framework for
  evaluating LLM safety mechanisms across mainly controlled substance queries,
  analyzing four major models' responses to systematically varied prompts. Our
  results reveal distinct safety profiles: Claude-3.5-sonnet demonstrated the
  most conservative approach with 73% refusals and 27% allowances, while Mistral
  attempted to answer 100% of queries. GPT-3.5-turbo showed moderate restriction
  with 10% refusals and 90% allowances, and Grok-2 registered 20% refusals and
  80% allowances. Testing prompt variation strategies revealed decreasing
  response consistency, from 85% with single prompts to 65% with five variations.
  This publicly available benchmark enables systematic evaluation of the critical
  balance between necessary safety restrictions and potential over-censorship of
  legitimate scientific inquiry, while providing a foundation for measuring
  progress in AI safety implementation. Chain-of-thought analysis reveals
  potential vulnerabilities in safety mechanisms, highlighting the complexity of
  implementing robust safeguards without unduly restricting desirable and valid
  scientific discourse.
paper_ai_summary: An open-source dataset and testing framework evaluate large language model safety mechanisms through controlled substance queries, revealing varying safety profiles and the need for balancing restrictions with legitimate scientific discourse.
paper_ai_keywords: large language models safety benchmarks open-source dataset testing framework safety mechanisms controlled substance queries Claude-3.5-sonnet Mistral GPT-3.5-turbo Grok-2 response consistency safety restrictions over-censorship chain-of-thought analysis safety vulnerabilities