TheAgentCompany: Benchmarking LLM Agents on Consequential Real World
  Tasks
We interact with computers on an everyday basis, be it in everyday life or
work, and many aspects of work can be done entirely with access to a computer
and the Internet. At the same time, thanks to improvements in large language
models (LLMs), there has also been a rapid development in AI agents that
interact with and affect change in their surrounding environments. But how
performant are AI agents at helping to accelerate or even autonomously perform
work-related tasks? The answer to this question has important implications for
both industry looking to adopt AI into their workflows, and for economic policy
to understand the effects that adoption of AI may have on the labor market. To
measure the progress of these LLM agents' performance on performing real-world
professional tasks, in this paper, we introduce TheAgentCompany, an extensible
benchmark for evaluating AI agents that interact with the world in similar ways
to those of a digital worker: by browsing the Web, writing code, running
programs, and communicating with other coworkers. We build a self-contained
environment with internal web sites and data that mimics a small software
company environment, and create a variety of tasks that may be performed by
workers in such a company. We test baseline agents powered by both closed
API-based and open-weights language models (LMs), and find that with the most
competitive agent, 24% of the tasks can be completed autonomously. This paints
a nuanced picture on task automation with LM agents -- in a setting simulating
a real workplace, a good portion of simpler tasks could be solved autonomously,
but more difficult long-horizon tasks are still beyond the reach of current
systems.
6763867f523f25389126b2aa
Frank F. Xu
6763867f523f25389126b2ab
Yufan Song
claimed_verified
2024-12-19T18:01:34.291Z
65f89af419e43f235cca48f9
/avatars/d1ff695f9a8ee0aeeff324fb7f07a811.svg
Yufan Song
user
yufansong
6763867f523f25389126b2ac
Boxuan Li
6763867f523f25389126b2ad
Yuxuan Tang
claimed_verified
2024-12-20T08:33:41.055Z
676482f298c1650813ed9f3b
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zZ8nc9mGFMrSmDZZu-KQD.png
Yuxuan Tang
user
tyxalan
6763867f523f25389126b2ae
Kritanjali Jain
admin_assigned
2024-12-20T09:39:30.381Z
6724d98943ef72ed99d4ec8d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Kud1lnWb8lfdoLuRW66x8.png
Kritanjali Jain
user
kritanjalijain
6763867f523f25389126b2af
Mengxue Bao
6763867f523f25389126b2b0
Zora Z. Wang
claimed_verified
2025-01-07T16:14:34.287Z
6356c77dd18276171735985f
https://cdn-avatars.huggingface.co/v1/production/uploads/6356c77dd18276171735985f/HYsRr1G13xgwVUoou7nFQ.jpeg
Zora Zhiruo Wang
user
zorazrw
6763867f523f25389126b2b1
Xuhui Zhou
admin_assigned
2024-12-20T09:40:09.108Z
61873839b1085ab638324e84
/avatars/a02d8067cd3a1cc4d9acff70cbc84078.svg
Xuhui Zhou
user
Xuhui
6763867f523f25389126b2b2
Zhitong Guo
6763867f523f25389126b2b3
Murong Cao
6763867f523f25389126b2b4
Mingyang Yang
6763867f523f25389126b2b5
Hao Yang Lu
6763867f523f25389126b2b6
Amaad Martin
admin_assigned
2024-12-20T09:41:00.558Z
64be22f865b648b2dfc0adb8
/avatars/e5875586d74c1421f0820df043dbb2c0.svg
Amaad Martin
user
AmaadMartin
6763867f523f25389126b2b7
Zhe Su
6763867f523f25389126b2b8
Leander Maben
admin_assigned
2024-12-20T09:41:15.121Z
67038c47c265fe2565e6f469
/avatars/cc46dd529e372cf12e29e5674c58ae82.svg
Leander Melroy Maben
user
leandermaben
6763867f523f25389126b2b9
Raj Mehta
6763867f523f25389126b2ba
Wayne Chi
admin_assigned
2024-12-20T09:41:49.616Z
6563bc8d6ef2a1d0f988445c
/avatars/19ed8a313c4fcb3ea617dd53772deeda.svg
Wayne Chi
user
waynechi
6763867f523f25389126b2bb
Lawrence Jang
admin_assigned
2024-12-19T18:08:00.787Z
664aebe829eadb3ab4e4ca3f
/avatars/548d5656e082c8959ac78b883f0805af.svg
Lawrence Jang
user
ljang0
6763867f523f25389126b2bc
Yiqing Xie
admin_assigned
2024-12-19T18:07:36.455Z
63433c7804d4ff28aeb957a2
https://cdn-avatars.huggingface.co/v1/production/uploads/63433c7804d4ff28aeb957a2/uGwqLh6O3CifrIOKqYsh3.jpeg
Yiqing Xie
user
yiqingx
6763867f523f25389126b2bd
Shuyan Zhou
6763867f523f25389126b2be
Graham Neubig
admin_assigned
2024-12-19T18:07:21.257Z
60de14638bedd2315529d43f
https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
Graham Neubig
user
gneubig
TheAgentCompany: Benchmarking LLM Agents on Consequential Real World
  Tasks
We interact with computers on an everyday basis, be it in everyday life or
work, and many aspects of work can be done entirely with access to a computer
and the Internet. At the same time, thanks to improvements in large language
models (LLMs), there has also been a rapid development in AI agents that
interact with and affect change in their surrounding environments. But how
performant are AI agents at helping to accelerate or even autonomously perform
work-related tasks? The answer to this question has important implications for
both industry looking to adopt AI into their workflows, and for economic policy
to understand the effects that adoption of AI may have on the labor market. To
measure the progress of these LLM agents' performance on performing real-world
professional tasks, in this paper, we introduce TheAgentCompany, an extensible
benchmark for evaluating AI agents that interact with the world in similar ways
to those of a digital worker: by browsing the Web, writing code, running
programs, and communicating with other coworkers. We build a self-contained
environment with internal web sites and data that mimics a small software
company environment, and create a variety of tasks that may be performed by
workers in such a company. We test baseline agents powered by both closed
API-based and open-weights language models (LMs), and find that with the most
competitive agent, 24% of the tasks can be completed autonomously. This paints
a nuanced picture on task automation with LM agents -- in a setting simulating
a real workplace, a good portion of simpler tasks could be solved autonomously,
but more difficult long-horizon tasks are still beyond the reach of current
systems.
A benchmark platform evaluates AI agents' autonomy in performing professional tasks in a simulated workplace environment, showing that a significant portion of simpler tasks can be automated but complex, long-term tasks remain challenging.
large language models AI agents TheAgentCompany benchmark digital worker web browsing code writing program running communication closed API-based open-weights language models task automation