title: Measuring AI Ability to Complete Long Tasks
summary: Despite rapid progress on AI benchmarks, the real-world meaning of benchmark
  performance remains unclear. To quantify the capabilities of AI systems in
  terms of human capabilities, we propose a new metric: 50%-task-completion time
  horizon. This is the time humans typically take to complete tasks that AI
  models can complete with 50% success rate. We first timed humans with relevant
  domain expertise on a combination of RE-Bench, HCAST, and 66 novel shorter
  tasks. On these tasks, current frontier AI models such as Claude 3.7 Sonnet
  have a 50% time horizon of around 50 minutes. Furthermore, frontier AI time
  horizon has been doubling approximately every seven months since 2019, though
  the trend may have accelerated in 2024. The increase in AI models' time
  horizons seems to be primarily driven by greater reliability and ability to
  adapt to mistakes, combined with better logical reasoning and tool use
  capabilities. We discuss the limitations of our results -- including their
  degree of external validity -- and the implications of increased autonomy for
  dangerous capabilities. If these results generalize to real-world software
  tasks, extrapolation of this trend predicts that within 5 years, AI systems
  will be capable of automating many software tasks that currently take humans a
  month.
paper_authors._id: 67da2e831bba0f73374fd5a0
paper_authors.name: Thomas Kwa
paper_authors._id: 67da2e831bba0f73374fd5a1
paper_authors.name: Ben West
paper_authors._id: 67da2e831bba0f73374fd5a2
paper_authors.name: Joel Becker
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:40:48.474Z
paper_authors._id: 664fe38dcd6b0c097982f9dc
paper_authors.avatarUrl: /avatars/00180e8352ea76c922bd86599af159f6.svg
paper_authors.fullname: Joel Becker
paper_authors.type: user
paper_authors.user: voxoid
paper_authors._id: 67da2e831bba0f73374fd5a3
paper_authors.name: Amy Deng
paper_authors._id: 67da2e831bba0f73374fd5a4
paper_authors.name: Katharyn Garcia
paper_authors._id: 67da2e831bba0f73374fd5a5
paper_authors.name: Max Hasin
paper_authors._id: 67da2e831bba0f73374fd5a6
paper_authors.name: Sami Jawhar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:41:05.417Z
paper_authors._id: 663bc1a595085055e96afdce
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/MK8s0IC7r93Tm4oQoPaqO.jpeg
paper_authors.fullname: Sami Jawhar
paper_authors.type: user
paper_authors.user: sjawhar
paper_authors._id: 67da2e831bba0f73374fd5a7
paper_authors.name: Megan Kinniment
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:41:11.957Z
paper_authors._id: 65fe059b2c739e076189fbb9
paper_authors.avatarUrl: /avatars/758063db87fbae4262ac12d41335990f.svg
paper_authors.fullname: Megan Kinniment
paper_authors.type: user
paper_authors.user: MeganKW
paper_authors._id: 67da2e831bba0f73374fd5a8
paper_authors.name: Nate Rush
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:41:18.725Z
paper_authors._id: 643458d8a4c9c55871a99894
paper_authors.avatarUrl: /avatars/df86144dafe543a973754235fa643e28.svg
paper_authors.fullname: Nate Rush
paper_authors.type: user
paper_authors.user: naterush
paper_authors._id: 67da2e831bba0f73374fd5a9
paper_authors.name: Sydney Von Arx
paper_authors._id: 67da2e831bba0f73374fd5aa
paper_authors.name: Ryan Bloom
paper_authors._id: 67da2e831bba0f73374fd5ab
paper_authors.name: Thomas Broadley
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:41:33.643Z
paper_authors._id: 664246cf273e54872591fde4
paper_authors.avatarUrl: /avatars/1f0e66467aa854de9d4fc938de4fac20.svg
paper_authors.fullname: Thomas Broadley
paper_authors.type: user
paper_authors.user: tbroadley
paper_authors._id: 67da2e831bba0f73374fd5ac
paper_authors.name: Haoxing Du
paper_authors._id: 67da2e831bba0f73374fd5ad
paper_authors.name: Brian Goodrich
paper_authors._id: 67da2e831bba0f73374fd5ae
paper_authors.name: Nikola Jurkovic
paper_authors._id: 67da2e831bba0f73374fd5af
paper_authors.name: Luke Harold Miles
paper_authors._id: 67da2e831bba0f73374fd5b0
paper_authors.name: Seraphina Nix
paper_authors._id: 67da2e831bba0f73374fd5b1
paper_authors.name: Tao Lin
paper_authors._id: 67da2e831bba0f73374fd5b2
paper_authors.name: Neev Parikh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:42:09.842Z
paper_authors._id: 653f71b92273676554d8cb14
paper_authors.avatarUrl: /avatars/64ef4c04aa3db8a3a5913598577c10bb.svg
paper_authors.fullname: Neev Parikh
paper_authors.type: user
paper_authors.user: neevparikh
paper_authors._id: 67da2e831bba0f73374fd5b3
paper_authors.name: David Rein
paper_authors._id: 67da2e831bba0f73374fd5b4
paper_authors.name: Lucas Jun Koba Sato
paper_authors._id: 67da2e831bba0f73374fd5b5
paper_authors.name: Hjalmar Wijk
paper_authors._id: 67da2e831bba0f73374fd5b6
paper_authors.name: Daniel M. Ziegler
paper_authors._id: 67da2e831bba0f73374fd5b7
paper_authors.name: Elizabeth Barnes
paper_authors._id: 67da2e831bba0f73374fd5b8
paper_authors.name: Lawrence Chan
paper_title: Measuring AI Ability to Complete Long Tasks
paper_summary: Despite rapid progress on AI benchmarks, the real-world meaning of benchmark
  performance remains unclear. To quantify the capabilities of AI systems in
  terms of human capabilities, we propose a new metric: 50%-task-completion time
  horizon. This is the time humans typically take to complete tasks that AI
  models can complete with 50% success rate. We first timed humans with relevant
  domain expertise on a combination of RE-Bench, HCAST, and 66 novel shorter
  tasks. On these tasks, current frontier AI models such as Claude 3.7 Sonnet
  have a 50% time horizon of around 50 minutes. Furthermore, frontier AI time
  horizon has been doubling approximately every seven months since 2019, though
  the trend may have accelerated in 2024. The increase in AI models' time
  horizons seems to be primarily driven by greater reliability and ability to
  adapt to mistakes, combined with better logical reasoning and tool use
  capabilities. We discuss the limitations of our results -- including their
  degree of external validity -- and the implications of increased autonomy for
  dangerous capabilities. If these results generalize to real-world software
  tasks, extrapolation of this trend predicts that within 5 years, AI systems
  will be capable of automating many software tasks that currently take humans a
  month.
paper_ai_summary: A new metric, 50%-task-completion time horizon, is proposed to quantify AI capabilities compared to human capabilities, with current frontier AI models showing a time horizon of around 50 minutes for tasks they complete with 50% success rate.
paper_ai_keywords: 50%-task-completion time horizon RE-Bench HCAST