CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models
<|reference_start|>CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models: We are releasing a new suite of security benchmarks for LLMs, CYBERSECEVAL 3, to continue the conversation on empirically measuring LLM cybersecurity risks and capabilities. CYBERSECEVAL 3 assesses 8 different risks across two broad categories: risk to third parties, and risk to application developers and end users. Compared to previous work, we add new areas focused on offensive security capabilities: automated social engineering, scaling manual offensive cyber operations, and autonomous offensive cyber operations. In this paper we discuss applying these benchmarks to the Llama 3 models and a suite of contemporaneous state-of-the-art LLMs, enabling us to contextualize risks both with and without mitigations in place.<|reference_end|>
@article{wan2024cyberseceval,
    title={CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and
  Capabilities in Large Language Models},
    author={Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James
  Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman,
  Stephanie Ding, Vlad Ionescu, Yue Li, Joshua Saxe},
    journal={arXiv preprint arXiv:2408.01605},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.01605},
    primaryClass={cs.CR cs.LG}
}