title: PaperBench: Evaluating AI's Ability to Replicate AI Research
summary: We introduce PaperBench, a benchmark evaluating the ability of AI agents to
  replicate state-of-the-art AI research. Agents must replicate 20 ICML 2024
  Spotlight and Oral papers from scratch, including understanding paper
  contributions, developing a codebase, and successfully executing experiments.
  For objective evaluation, we develop rubrics that hierarchically decompose each
  replication task into smaller sub-tasks with clear grading criteria. In total,
  PaperBench contains 8,316 individually gradable tasks. Rubrics are co-developed
  with the author(s) of each ICML paper for accuracy and realism. To enable
  scalable evaluation, we also develop an LLM-based judge to automatically grade
  replication attempts against rubrics, and assess our judge's performance by
  creating a separate benchmark for judges. We evaluate several frontier models
  on PaperBench, finding that the best-performing tested agent, Claude 3.5 Sonnet
  (New) with open-source scaffolding, achieves an average replication score of
  21.0\%. Finally, we recruit top ML PhDs to attempt a subset of PaperBench,
  finding that models do not yet outperform the human baseline. We
  https://github.com/openai/preparedness{open-source our code} to
  facilitate future research in understanding the AI engineering capabilities of
  AI agents.
paper_authors._id: 67edf3d579018bf61e050435
paper_authors.name: Giulio Starace
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T13:57:27.629Z
paper_authors._id: 627b8bc83974b0ed6b28db67
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665158475637-627b8bc83974b0ed6b28db67.jpeg
paper_authors.fullname: Giulio Starace
paper_authors.type: user
paper_authors.user: thesofakillers
paper_authors._id: 67edf3d579018bf61e050436
paper_authors.name: Oliver Jaffe
paper_authors._id: 67edf3d579018bf61e050437
paper_authors.name: Dane Sherburn
paper_authors._id: 67edf3d579018bf61e050438
paper_authors.name: James Aung
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T13:57:49.748Z
paper_authors._id: 6316bd602fd6e930e429b362
paper_authors.avatarUrl: /avatars/28abc3ad3a00811da404f5d88dd25a69.svg
paper_authors.fullname: James Aung
paper_authors.type: user
paper_authors.user: james-aung
paper_authors._id: 67edf3d579018bf61e050439
paper_authors.name: Jun Shern Chan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T13:58:00.906Z
paper_authors._id: 60db178afb288e95534e7909
paper_authors.avatarUrl: /avatars/54a656506a66d06aeb3556c88a9a072a.svg
paper_authors.fullname: Chan Jun Shern
paper_authors.type: user
paper_authors.user: junshern
paper_authors._id: 67edf3d579018bf61e05043a
paper_authors.name: Leon Maksin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T13:58:13.107Z
paper_authors._id: 632c9a1c1d303f5f9ad0685c
paper_authors.avatarUrl: /avatars/d8828064fe0d89efed7594c5c9747078.svg
paper_authors.fullname: M
paper_authors.type: user
paper_authors.user: LeonM
paper_authors._id: 67edf3d579018bf61e05043b
paper_authors.name: Rachel Dias
paper_authors._id: 67edf3d579018bf61e05043c
paper_authors.name: Evan Mays
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T13:58:23.365Z
paper_authors._id: 636d9548da13cd91f922e0d0
paper_authors.avatarUrl: /avatars/488f6a4b12f20809d3a4c3d40ae82b53.svg
paper_authors.fullname: Evan Mays
paper_authors.type: user
paper_authors.user: evanmays
paper_authors._id: 67edf3d579018bf61e05043d
paper_authors.name: Benjamin Kinsella
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T13:58:29.583Z
paper_authors._id: 65397cb82a101a151da4836b
paper_authors.avatarUrl: /avatars/dd9e03be4e06fd41b82cce97c9ca2d21.svg
paper_authors.fullname: Benjamin Kinsella
paper_authors.type: user
paper_authors.user: bjk127
paper_authors._id: 67edf3d579018bf61e05043e
paper_authors.name: Wyatt Thompson
paper_authors._id: 67edf3d579018bf61e05043f
paper_authors.name: Johannes Heidecke
paper_authors._id: 67edf3d579018bf61e050440
paper_authors.name: Amelia Glaese
paper_authors._id: 67edf3d579018bf61e050441
paper_authors.name: Tejal Patwardhan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T13:58:46.470Z
paper_authors._id: 64d4f504887f55fb6eedec74
paper_authors.avatarUrl: /avatars/054fb826890adcb330f0e4cbca3ef7c4.svg
paper_authors.fullname: Tejal Patwardhan
paper_authors.type: user
paper_authors.user: tejalp
paper_title: PaperBench: Evaluating AI's Ability to Replicate AI Research
paper_summary: We introduce PaperBench, a benchmark evaluating the ability of AI agents to
  replicate state-of-the-art AI research. Agents must replicate 20 ICML 2024
  Spotlight and Oral papers from scratch, including understanding paper
  contributions, developing a codebase, and successfully executing experiments.
  For objective evaluation, we develop rubrics that hierarchically decompose each
  replication task into smaller sub-tasks with clear grading criteria. In total,
  PaperBench contains 8,316 individually gradable tasks. Rubrics are co-developed
  with the author(s) of each ICML paper for accuracy and realism. To enable
  scalable evaluation, we also develop an LLM-based judge to automatically grade
  replication attempts against rubrics, and assess our judge's performance by
  creating a separate benchmark for judges. We evaluate several frontier models
  on PaperBench, finding that the best-performing tested agent, Claude 3.5 Sonnet
  (New) with open-source scaffolding, achieves an average replication score of
  21.0\%. Finally, we recruit top ML PhDs to attempt a subset of PaperBench,
  finding that models do not yet outperform the human baseline. We
  https://github.com/openai/preparedness{open-source our code} to
  facilitate future research in understanding the AI engineering capabilities of
  AI agents.
paper_ai_summary: PaperBench evaluates AI agents' ability to replicate state-of-the-art AI research by decomposing replication tasks into graded sub-tasks, using both LLM-based and human judges to assess performance.
paper_ai_keywords: benchmark ICML 2024 rubrics LLM-based judge agent replication Claude 3.5 Sonnet ML PhDs AI engineering capabilities