Measuring AI Ability to Complete Long Tasks
Despite rapid progress on AI benchmarks, the real-world meaning of benchmark
performance remains unclear. To quantify the capabilities of AI systems in
terms of human capabilities, we propose a new metric: 50%-task-completion time
horizon. This is the time humans typically take to complete tasks that AI
models can complete with 50% success rate. We first timed humans with relevant
domain expertise on a combination of RE-Bench, HCAST, and 66 novel shorter
tasks. On these tasks, current frontier AI models such as Claude 3.7 Sonnet
have a 50% time horizon of around 50 minutes. Furthermore, frontier AI time
horizon has been doubling approximately every seven months since 2019, though
the trend may have accelerated in 2024. The increase in AI models' time
horizons seems to be primarily driven by greater reliability and ability to
adapt to mistakes, combined with better logical reasoning and tool use
capabilities. We discuss the limitations of our results -- including their
degree of external validity -- and the implications of increased autonomy for
dangerous capabilities. If these results generalize to real-world software
tasks, extrapolation of this trend predicts that within 5 years, AI systems
will be capable of automating many software tasks that currently take humans a
month.
67da2e831bba0f73374fd5a0
Thomas Kwa
67da2e831bba0f73374fd5a1
Ben West
67da2e831bba0f73374fd5a2
Joel Becker
admin_assigned
2025-03-19T13:40:48.474Z
664fe38dcd6b0c097982f9dc
/avatars/00180e8352ea76c922bd86599af159f6.svg
Joel Becker
user
voxoid
67da2e831bba0f73374fd5a3
Amy Deng
67da2e831bba0f73374fd5a4
Katharyn Garcia
67da2e831bba0f73374fd5a5
Max Hasin
67da2e831bba0f73374fd5a6
Sami Jawhar
admin_assigned
2025-03-19T13:41:05.417Z
663bc1a595085055e96afdce
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/MK8s0IC7r93Tm4oQoPaqO.jpeg
Sami Jawhar
user
sjawhar
67da2e831bba0f73374fd5a7
Megan Kinniment
admin_assigned
2025-03-19T13:41:11.957Z
65fe059b2c739e076189fbb9
/avatars/758063db87fbae4262ac12d41335990f.svg
Megan Kinniment
user
MeganKW
67da2e831bba0f73374fd5a8
Nate Rush
admin_assigned
2025-03-19T13:41:18.725Z
643458d8a4c9c55871a99894
/avatars/df86144dafe543a973754235fa643e28.svg
Nate Rush
user
naterush
67da2e831bba0f73374fd5a9
Sydney Von Arx
67da2e831bba0f73374fd5aa
Ryan Bloom
67da2e831bba0f73374fd5ab
Thomas Broadley
admin_assigned
2025-03-19T13:41:33.643Z
664246cf273e54872591fde4
/avatars/1f0e66467aa854de9d4fc938de4fac20.svg
Thomas Broadley
user
tbroadley
67da2e831bba0f73374fd5ac
Haoxing Du
67da2e831bba0f73374fd5ad
Brian Goodrich
67da2e831bba0f73374fd5ae
Nikola Jurkovic
67da2e831bba0f73374fd5af
Luke Harold Miles
67da2e831bba0f73374fd5b0
Seraphina Nix
67da2e831bba0f73374fd5b1
Tao Lin
67da2e831bba0f73374fd5b2
Neev Parikh
admin_assigned
2025-03-19T13:42:09.842Z
653f71b92273676554d8cb14
/avatars/64ef4c04aa3db8a3a5913598577c10bb.svg
Neev Parikh
user
neevparikh
67da2e831bba0f73374fd5b3
David Rein
67da2e831bba0f73374fd5b4
Lucas Jun Koba Sato
67da2e831bba0f73374fd5b5
Hjalmar Wijk
67da2e831bba0f73374fd5b6
Daniel M. Ziegler
67da2e831bba0f73374fd5b7
Elizabeth Barnes
67da2e831bba0f73374fd5b8
Lawrence Chan
Measuring AI Ability to Complete Long Tasks
Despite rapid progress on AI benchmarks, the real-world meaning of benchmark
performance remains unclear. To quantify the capabilities of AI systems in
terms of human capabilities, we propose a new metric: 50%-task-completion time
horizon. This is the time humans typically take to complete tasks that AI
models can complete with 50% success rate. We first timed humans with relevant
domain expertise on a combination of RE-Bench, HCAST, and 66 novel shorter
tasks. On these tasks, current frontier AI models such as Claude 3.7 Sonnet
have a 50% time horizon of around 50 minutes. Furthermore, frontier AI time
horizon has been doubling approximately every seven months since 2019, though
the trend may have accelerated in 2024. The increase in AI models' time
horizons seems to be primarily driven by greater reliability and ability to
adapt to mistakes, combined with better logical reasoning and tool use
capabilities. We discuss the limitations of our results -- including their
degree of external validity -- and the implications of increased autonomy for
dangerous capabilities. If these results generalize to real-world software
tasks, extrapolation of this trend predicts that within 5 years, AI systems
will be capable of automating many software tasks that currently take humans a
month.
A new metric, 50%-task-completion time horizon, is proposed to quantify AI capabilities compared to human capabilities, with current frontier AI models showing a time horizon of around 50 minutes for tasks they complete with 50% success rate.
50%-task-completion time horizon RE-Bench HCAST