Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk
  of Language Models
Language Model (LM) agents for cybersecurity that are capable of autonomously
identifying vulnerabilities and executing exploits have the potential to cause
real-world impact. Policymakers, model providers, and other researchers in the
AI and cybersecurity communities are interested in quantifying the capabilities
of such agents to help mitigate cyberrisk and investigate opportunities for
penetration testing. Toward that end, we introduce Cybench, a framework for
specifying cybersecurity tasks and evaluating agents on those tasks. We include
40 professional-level Capture the Flag (CTF) tasks from 4 distinct CTF
competitions, chosen to be recent, meaningful, and spanning a wide range of
difficulties. Each task includes its own description, starter files, and is
initialized in an environment where an agent can execute bash commands and
observe outputs. Since many tasks are beyond the capabilities of existing LM
agents, we introduce subtasks, which break down a task into intermediary steps
for more gradated evaluation; we add subtasks for 17 of the 40 tasks. To
evaluate agent capabilities, we construct a cybersecurity agent and evaluate 7
models: GPT-4o, Claude 3 Opus, Claude 3.5 Sonnet, Mixtral 8x22b Instruct,
Gemini 1.5 Pro, Llama 3 70B Chat, and Llama 3.1 405B Instruct. Without
guidance, we find that agents are able to solve only the easiest complete tasks
that took human teams up to 11 minutes to solve, with Claude 3.5 Sonnet and
GPT-4o having the highest success rates. Finally, subtasks provide more signal
for measuring performance compared to unguided runs, with models achieving a
3.2\% higher success rate on complete tasks with subtask-guidance than without
subtask-guidance. All code and data are publicly available at
https://cybench.github.io
66c411b626fef28c3e050122
Andy K. Zhang
66c411b626fef28c3e050123
Neil Perry
66c411b626fef28c3e050124
Riya Dulepet
66c411b626fef28c3e050125
Eliot Jones
claimed_verified
2024-10-31T09:55:32.645Z
66214936126ebc0deb501a87
/avatars/e038e84f58c8577425df327a06b801d0.svg
Eliot Jones
user
eliotj
66c411b626fef28c3e050126
Justin W. Lin
claimed_verified
2025-04-16T09:42:42.124Z
605bbbaa2bd7bebcc260c2ab
/avatars/649ab95509e2c09d54fe5e46b5f41d28.svg
Justin W Lin
user
justinwlin
66c411b626fef28c3e050127
Joey Ji
66c411b626fef28c3e050128
Celeste Menders
66c411b626fef28c3e050129
Gashon Hussein
66c411b626fef28c3e05012a
Samantha Liu
66c411b626fef28c3e05012b
Donovan Jasper
66c411b626fef28c3e05012c
Pura Peetathawatchai
66c411b626fef28c3e05012d
Ari Glenn
66c411b626fef28c3e05012e
Vikram Sivashankar
66c411b626fef28c3e05012f
Daniel Zamoshchin
66c411b626fef28c3e050130
Leo Glikbarg
66c411b626fef28c3e050131
Derek Askaryar
66c411b626fef28c3e050132
Mike Yang
66c411b626fef28c3e050133
Teddy Zhang
66c411b626fef28c3e050134
Rishi Alluri
66c411b626fef28c3e050135
Nathan Tran
66c411b626fef28c3e050136
Rinnara Sangpisit
66c411b626fef28c3e050137
Polycarpos Yiorkadjis
66c411b626fef28c3e050138
Kenny Osele
66c411b626fef28c3e050139
Gautham Raghupathi
66c411b626fef28c3e05013a
Dan Boneh
66c411b626fef28c3e05013b
Daniel E. Ho
66c411b626fef28c3e05013c
Percy Liang
Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk
  of Language Models
Language Model (LM) agents for cybersecurity that are capable of autonomously
identifying vulnerabilities and executing exploits have the potential to cause
real-world impact. Policymakers, model providers, and other researchers in the
AI and cybersecurity communities are interested in quantifying the capabilities
of such agents to help mitigate cyberrisk and investigate opportunities for
penetration testing. Toward that end, we introduce Cybench, a framework for
specifying cybersecurity tasks and evaluating agents on those tasks. We include
40 professional-level Capture the Flag (CTF) tasks from 4 distinct CTF
competitions, chosen to be recent, meaningful, and spanning a wide range of
difficulties. Each task includes its own description, starter files, and is
initialized in an environment where an agent can execute bash commands and
observe outputs. Since many tasks are beyond the capabilities of existing LM
agents, we introduce subtasks, which break down a task into intermediary steps
for more gradated evaluation; we add subtasks for 17 of the 40 tasks. To
evaluate agent capabilities, we construct a cybersecurity agent and evaluate 7
models: GPT-4o, Claude 3 Opus, Claude 3.5 Sonnet, Mixtral 8x22b Instruct,
Gemini 1.5 Pro, Llama 3 70B Chat, and Llama 3.1 405B Instruct. Without
guidance, we find that agents are able to solve only the easiest complete tasks
that took human teams up to 11 minutes to solve, with Claude 3.5 Sonnet and
GPT-4o having the highest success rates. Finally, subtasks provide more signal
for measuring performance compared to unguided runs, with models achieving a
3.2\% higher success rate on complete tasks with subtask-guidance than without
subtask-guidance. All code and data are publicly available at
https://cybench.github.io
Cybench evaluates language model agents for cybersecurity tasks, including challenges from CTF competitions, using both complete tasks and subtasks to assess performance.
Capture the Flag CTF subtasks cybersecurity agent GPT-4o Claude 3 Opus Claude 3.5 Sonnet Mixtral 8x22b Instruct Gemini 1.5 Pro Llama 3 70B Chat Llama 3.1 405B Instruct