title: Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk
    of Language Models
summary: Language Model (LM) agents for cybersecurity that are capable of autonomously
  identifying vulnerabilities and executing exploits have the potential to cause
  real-world impact. Policymakers, model providers, and other researchers in the
  AI and cybersecurity communities are interested in quantifying the capabilities
  of such agents to help mitigate cyberrisk and investigate opportunities for
  penetration testing. Toward that end, we introduce Cybench, a framework for
  specifying cybersecurity tasks and evaluating agents on those tasks. We include
  40 professional-level Capture the Flag (CTF) tasks from 4 distinct CTF
  competitions, chosen to be recent, meaningful, and spanning a wide range of
  difficulties. Each task includes its own description, starter files, and is
  initialized in an environment where an agent can execute bash commands and
  observe outputs. Since many tasks are beyond the capabilities of existing LM
  agents, we introduce subtasks, which break down a task into intermediary steps
  for more gradated evaluation; we add subtasks for 17 of the 40 tasks. To
  evaluate agent capabilities, we construct a cybersecurity agent and evaluate 7
  models: GPT-4o, Claude 3 Opus, Claude 3.5 Sonnet, Mixtral 8x22b Instruct,
  Gemini 1.5 Pro, Llama 3 70B Chat, and Llama 3.1 405B Instruct. Without
  guidance, we find that agents are able to solve only the easiest complete tasks
  that took human teams up to 11 minutes to solve, with Claude 3.5 Sonnet and
  GPT-4o having the highest success rates. Finally, subtasks provide more signal
  for measuring performance compared to unguided runs, with models achieving a
  3.2\% higher success rate on complete tasks with subtask-guidance than without
  subtask-guidance. All code and data are publicly available at
  https://cybench.github.io
paper_authors._id: 66c411b626fef28c3e050122
paper_authors.name: Andy K. Zhang
paper_authors._id: 66c411b626fef28c3e050123
paper_authors.name: Neil Perry
paper_authors._id: 66c411b626fef28c3e050124
paper_authors.name: Riya Dulepet
paper_authors._id: 66c411b626fef28c3e050125
paper_authors.name: Eliot Jones
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-31T09:55:32.645Z
paper_authors._id: 66214936126ebc0deb501a87
paper_authors.avatarUrl: /avatars/e038e84f58c8577425df327a06b801d0.svg
paper_authors.fullname: Eliot Jones
paper_authors.type: user
paper_authors.user: eliotj
paper_authors._id: 66c411b626fef28c3e050126
paper_authors.name: Justin W. Lin
paper_authors._id: 66c411b626fef28c3e050127
paper_authors.name: Joey Ji
paper_authors._id: 66c411b626fef28c3e050128
paper_authors.name: Celeste Menders
paper_authors._id: 66c411b626fef28c3e050129
paper_authors.name: Gashon Hussein
paper_authors._id: 66c411b626fef28c3e05012a
paper_authors.name: Samantha Liu
paper_authors._id: 66c411b626fef28c3e05012b
paper_authors.name: Donovan Jasper
paper_authors._id: 66c411b626fef28c3e05012c
paper_authors.name: Pura Peetathawatchai
paper_authors._id: 66c411b626fef28c3e05012d
paper_authors.name: Ari Glenn
paper_authors._id: 66c411b626fef28c3e05012e
paper_authors.name: Vikram Sivashankar
paper_authors._id: 66c411b626fef28c3e05012f
paper_authors.name: Daniel Zamoshchin
paper_authors._id: 66c411b626fef28c3e050130
paper_authors.name: Leo Glikbarg
paper_authors._id: 66c411b626fef28c3e050131
paper_authors.name: Derek Askaryar
paper_authors._id: 66c411b626fef28c3e050132
paper_authors.name: Mike Yang
paper_authors._id: 66c411b626fef28c3e050133
paper_authors.name: Teddy Zhang
paper_authors._id: 66c411b626fef28c3e050134
paper_authors.name: Rishi Alluri
paper_authors._id: 66c411b626fef28c3e050135
paper_authors.name: Nathan Tran
paper_authors._id: 66c411b626fef28c3e050136
paper_authors.name: Rinnara Sangpisit
paper_authors._id: 66c411b626fef28c3e050137
paper_authors.name: Polycarpos Yiorkadjis
paper_authors._id: 66c411b626fef28c3e050138
paper_authors.name: Kenny Osele
paper_authors._id: 66c411b626fef28c3e050139
paper_authors.name: Gautham Raghupathi
paper_authors._id: 66c411b626fef28c3e05013a
paper_authors.name: Dan Boneh
paper_authors._id: 66c411b626fef28c3e05013b
paper_authors.name: Daniel E. Ho
paper_authors._id: 66c411b626fef28c3e05013c
paper_authors.name: Percy Liang
paper_title: Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk
    of Language Models
paper_summary: Language Model (LM) agents for cybersecurity that are capable of autonomously
  identifying vulnerabilities and executing exploits have the potential to cause
  real-world impact. Policymakers, model providers, and other researchers in the
  AI and cybersecurity communities are interested in quantifying the capabilities
  of such agents to help mitigate cyberrisk and investigate opportunities for
  penetration testing. Toward that end, we introduce Cybench, a framework for
  specifying cybersecurity tasks and evaluating agents on those tasks. We include
  40 professional-level Capture the Flag (CTF) tasks from 4 distinct CTF
  competitions, chosen to be recent, meaningful, and spanning a wide range of
  difficulties. Each task includes its own description, starter files, and is
  initialized in an environment where an agent can execute bash commands and
  observe outputs. Since many tasks are beyond the capabilities of existing LM
  agents, we introduce subtasks, which break down a task into intermediary steps
  for more gradated evaluation; we add subtasks for 17 of the 40 tasks. To
  evaluate agent capabilities, we construct a cybersecurity agent and evaluate 7
  models: GPT-4o, Claude 3 Opus, Claude 3.5 Sonnet, Mixtral 8x22b Instruct,
  Gemini 1.5 Pro, Llama 3 70B Chat, and Llama 3.1 405B Instruct. Without
  guidance, we find that agents are able to solve only the easiest complete tasks
  that took human teams up to 11 minutes to solve, with Claude 3.5 Sonnet and
  GPT-4o having the highest success rates. Finally, subtasks provide more signal
  for measuring performance compared to unguided runs, with models achieving a
  3.2\% higher success rate on complete tasks with subtask-guidance than without
  subtask-guidance. All code and data are publicly available at
  https://cybench.github.io
paper_ai_keywords: Capture the Flag (CTF) cybersecurity tasks cybersecurity agent subtasks GPT-4o Claude 3 Opus Claude 3.5 Sonnet Mixtral 8x22b Instruct Gemini 1.5 Pro Llama 3 70B Chat Llama 3.1 405B Instruct subtask-guidance