garak: A Framework for Security Probing Large Language Models
<|reference_start|>garak: A Framework for Security Probing Large Language Models: As Large Language Models (LLMs) are deployed and integrated into thousands of applications, the need for scalable evaluation of how models respond to adversarial attacks grows rapidly. However, LLM security is a moving target: models produce unpredictable output, are constantly updated, and the potential adversary is highly diverse: anyone with access to the internet and a decent command of natural language. Further, what constitutes a security weak in one context may not be an issue in a different context; one-fits-all guardrails remain theoretical. In this paper, we argue that it is time to rethink what constitutes ``LLM security'', and pursue a holistic approach to LLM security evaluation, where exploration and discovery of issues are central. To this end, this paper introduces garak (Generative AI Red-teaming and Assessment Kit), a framework which can be used to discover and identify vulnerabilities in a target LLM or dialog system. garak probes an LLM in a structured fashion to discover potential vulnerabilities. The outputs of the framework describe a target model's weaknesses, contribute to an informed discussion of what composes vulnerabilities in unique contexts, and can inform alignment and policy discussions for LLM deployment.<|reference_end|>
@article{derczynski2024garak:,
    title={garak: A Framework for Security Probing Large Language Models},
    author={Leon Derczynski, Erick Galinkin, Jeffrey Martin, Subho Majumdar, Nanna
  Inie},
    journal={arXiv preprint arXiv:2406.11036},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.11036},
    primaryClass={cs.CL cs.CR}
}