title: TheAgentCompany: Benchmarking LLM Agents on Consequential Real World
    Tasks
summary: We interact with computers on an everyday basis, be it in everyday life or
  work, and many aspects of work can be done entirely with access to a computer
  and the Internet. At the same time, thanks to improvements in large language
  models (LLMs), there has also been a rapid development in AI agents that
  interact with and affect change in their surrounding environments. But how
  performant are AI agents at helping to accelerate or even autonomously perform
  work-related tasks? The answer to this question has important implications for
  both industry looking to adopt AI into their workflows, and for economic policy
  to understand the effects that adoption of AI may have on the labor market. To
  measure the progress of these LLM agents' performance on performing real-world
  professional tasks, in this paper, we introduce TheAgentCompany, an extensible
  benchmark for evaluating AI agents that interact with the world in similar ways
  to those of a digital worker: by browsing the Web, writing code, running
  programs, and communicating with other coworkers. We build a self-contained
  environment with internal web sites and data that mimics a small software
  company environment, and create a variety of tasks that may be performed by
  workers in such a company. We test baseline agents powered by both closed
  API-based and open-weights language models (LMs), and find that with the most
  competitive agent, 24% of the tasks can be completed autonomously. This paints
  a nuanced picture on task automation with LM agents -- in a setting simulating
  a real workplace, a good portion of simpler tasks could be solved autonomously,
  but more difficult long-horizon tasks are still beyond the reach of current
  systems.
paper_authors._id: 6763867f523f25389126b2aa
paper_authors.name: Frank F. Xu
paper_authors._id: 6763867f523f25389126b2ab
paper_authors.name: Yufan Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:01:34.291Z
paper_authors._id: 65f89af419e43f235cca48f9
paper_authors.avatarUrl: /avatars/d1ff695f9a8ee0aeeff324fb7f07a811.svg
paper_authors.fullname: Yufan Song
paper_authors.type: user
paper_authors.user: yufansong
paper_authors._id: 6763867f523f25389126b2ac
paper_authors.name: Boxuan Li
paper_authors._id: 6763867f523f25389126b2ad
paper_authors.name: Yuxuan Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:33:41.055Z
paper_authors._id: 676482f298c1650813ed9f3b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zZ8nc9mGFMrSmDZZu-KQD.png
paper_authors.fullname: Yuxuan Tang
paper_authors.type: user
paper_authors.user: tyxalan
paper_authors._id: 6763867f523f25389126b2ae
paper_authors.name: Kritanjali Jain
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:39:30.381Z
paper_authors._id: 6724d98943ef72ed99d4ec8d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Kud1lnWb8lfdoLuRW66x8.png
paper_authors.fullname: Kritanjali Jain
paper_authors.type: user
paper_authors.user: kritanjalijain
paper_authors._id: 6763867f523f25389126b2af
paper_authors.name: Mengxue Bao
paper_authors._id: 6763867f523f25389126b2b0
paper_authors.name: Zora Z. Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T16:14:34.287Z
paper_authors._id: 6356c77dd18276171735985f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6356c77dd18276171735985f/HYsRr1G13xgwVUoou7nFQ.jpeg
paper_authors.fullname: Zora Zhiruo Wang
paper_authors.type: user
paper_authors.user: zorazrw
paper_authors._id: 6763867f523f25389126b2b1
paper_authors.name: Xuhui Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:40:09.108Z
paper_authors._id: 61873839b1085ab638324e84
paper_authors.avatarUrl: /avatars/a02d8067cd3a1cc4d9acff70cbc84078.svg
paper_authors.fullname: Xuhui Zhou
paper_authors.type: user
paper_authors.user: Xuhui
paper_authors._id: 6763867f523f25389126b2b2
paper_authors.name: Zhitong Guo
paper_authors._id: 6763867f523f25389126b2b3
paper_authors.name: Murong Cao
paper_authors._id: 6763867f523f25389126b2b4
paper_authors.name: Mingyang Yang
paper_authors._id: 6763867f523f25389126b2b5
paper_authors.name: Hao Yang Lu
paper_authors._id: 6763867f523f25389126b2b6
paper_authors.name: Amaad Martin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:41:00.558Z
paper_authors._id: 64be22f865b648b2dfc0adb8
paper_authors.avatarUrl: /avatars/e5875586d74c1421f0820df043dbb2c0.svg
paper_authors.fullname: Amaad Martin
paper_authors.type: user
paper_authors.user: AmaadMartin
paper_authors._id: 6763867f523f25389126b2b7
paper_authors.name: Zhe Su
paper_authors._id: 6763867f523f25389126b2b8
paper_authors.name: Leander Maben
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:41:15.121Z
paper_authors._id: 67038c47c265fe2565e6f469
paper_authors.avatarUrl: /avatars/cc46dd529e372cf12e29e5674c58ae82.svg
paper_authors.fullname: Leander Melroy Maben
paper_authors.type: user
paper_authors.user: leandermaben
paper_authors._id: 6763867f523f25389126b2b9
paper_authors.name: Raj Mehta
paper_authors._id: 6763867f523f25389126b2ba
paper_authors.name: Wayne Chi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:41:49.616Z
paper_authors._id: 6563bc8d6ef2a1d0f988445c
paper_authors.avatarUrl: /avatars/19ed8a313c4fcb3ea617dd53772deeda.svg
paper_authors.fullname: Wayne Chi
paper_authors.type: user
paper_authors.user: waynechi
paper_authors._id: 6763867f523f25389126b2bb
paper_authors.name: Lawrence Jang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-19T18:08:00.787Z
paper_authors._id: 664aebe829eadb3ab4e4ca3f
paper_authors.avatarUrl: /avatars/548d5656e082c8959ac78b883f0805af.svg
paper_authors.fullname: Lawrence Jang
paper_authors.type: user
paper_authors.user: ljang0
paper_authors._id: 6763867f523f25389126b2bc
paper_authors.name: Yiqing Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-19T18:07:36.455Z
paper_authors._id: 63433c7804d4ff28aeb957a2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63433c7804d4ff28aeb957a2/uGwqLh6O3CifrIOKqYsh3.jpeg
paper_authors.fullname: Yiqing Xie
paper_authors.type: user
paper_authors.user: yiqingx
paper_authors._id: 6763867f523f25389126b2bd
paper_authors.name: Shuyan Zhou
paper_authors._id: 6763867f523f25389126b2be
paper_authors.name: Graham Neubig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-19T18:07:21.257Z
paper_authors._id: 60de14638bedd2315529d43f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
paper_authors.fullname: Graham Neubig
paper_authors.type: user
paper_authors.user: gneubig
paper_title: TheAgentCompany: Benchmarking LLM Agents on Consequential Real World
    Tasks
paper_summary: We interact with computers on an everyday basis, be it in everyday life or
  work, and many aspects of work can be done entirely with access to a computer
  and the Internet. At the same time, thanks to improvements in large language
  models (LLMs), there has also been a rapid development in AI agents that
  interact with and affect change in their surrounding environments. But how
  performant are AI agents at helping to accelerate or even autonomously perform
  work-related tasks? The answer to this question has important implications for
  both industry looking to adopt AI into their workflows, and for economic policy
  to understand the effects that adoption of AI may have on the labor market. To
  measure the progress of these LLM agents' performance on performing real-world
  professional tasks, in this paper, we introduce TheAgentCompany, an extensible
  benchmark for evaluating AI agents that interact with the world in similar ways
  to those of a digital worker: by browsing the Web, writing code, running
  programs, and communicating with other coworkers. We build a self-contained
  environment with internal web sites and data that mimics a small software
  company environment, and create a variety of tasks that may be performed by
  workers in such a company. We test baseline agents powered by both closed
  API-based and open-weights language models (LMs), and find that with the most
  competitive agent, 24% of the tasks can be completed autonomously. This paints
  a nuanced picture on task automation with LM agents -- in a setting simulating
  a real workplace, a good portion of simpler tasks could be solved autonomously,
  but more difficult long-horizon tasks are still beyond the reach of current
  systems.
paper_ai_summary: A benchmark platform evaluates AI agents' autonomy in performing professional tasks in a simulated workplace environment, showing that a significant portion of simpler tasks can be automated but complex, long-term tasks remain challenging.
paper_ai_keywords: large language models AI agents TheAgentCompany benchmark digital worker web browsing code writing program running communication closed API-based open-weights language models task automation