title: WebArena: A Realistic Web Environment for Building Autonomous Agents
summary: With generative AI advances, the exciting potential for autonomous agents to
  manage daily tasks via natural language commands has emerged. However, cur rent
  agents are primarily created and tested in simplified synthetic environments,
  substantially limiting real-world scenario representation. In this paper, we
  build an environment for agent command and control that is highly realistic and
  reproducible. Specifically, we focus on agents that perform tasks on websites,
  and we create an environment with fully functional websites from four common
  domains: e-commerce, social forum discussions, collaborative software
  development, and content management. Our environment is enriched with tools
  (e.g., a map) and external knowledge bases (e.g., user manuals) to encourage
  human-like task-solving. Building upon our environment, we release a set of
  benchmark tasks focusing on evaluating the functional correctness of task
  completions. The tasks in our benchmark are diverse, long-horizon, and are
  designed to emulate tasks that humans routinely perform on the internet. We
  design and implement several autonomous agents, integrating recent techniques
  such as reasoning before acting. The results demonstrate that solving complex
  tasks is challenging: our best GPT-4-based agent only achieves an end-to-end
  task success rate of 10.59%. These results highlight the need for further
  development of robust agents, that current state-of-the-art LMs are far from
  perfect performance in these real-life tasks, and that WebArena can be used to
  measure such progress. Our code, data, environment reproduction resources, and
  video demonstrations are publicly available at https://webarena.dev/.
paper_authors._id: 64c1c37bec3c6181351f748c
paper_authors.name: Shuyan Zhou
paper_authors._id: 64c1c37bec3c6181351f748d
paper_authors.name: Frank F. Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-27T10:42:46.834Z
paper_authors._id: 61ac0a0f57307cf7ae3a35bb
paper_authors.avatarUrl: /avatars/c3b00cf275b2805861f5b020c9d00639.svg
paper_authors.fullname: Frank Xu
paper_authors.type: user
paper_authors.user: frankxu
paper_authors._id: 64c1c37bec3c6181351f748e
paper_authors.name: Hao Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-14T12:31:41.475Z
paper_authors._id: 61aa376688c20eebf1e8deb3
paper_authors.avatarUrl: /avatars/7c11dcb232c73547d7d87834be287822.svg
paper_authors.fullname: Hao Zhu
paper_authors.type: user
paper_authors.user: ProKil
paper_authors._id: 64c1c37bec3c6181351f748f
paper_authors.name: Xuhui Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-27T10:44:28.825Z
paper_authors._id: 61873839b1085ab638324e84
paper_authors.avatarUrl: /avatars/a02d8067cd3a1cc4d9acff70cbc84078.svg
paper_authors.fullname: Xuhui Zhou
paper_authors.type: user
paper_authors.user: Xuhui
paper_authors._id: 64c1c37bec3c6181351f7490
paper_authors.name: Robert Lo
paper_authors._id: 64c1c37bec3c6181351f7491
paper_authors.name: Abishek Sridhar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-27T10:45:44.755Z
paper_authors._id: 63bc400f141c7d395c4a9383
paper_authors.avatarUrl: /avatars/3a494e3aa55096c506ebaecb3dee0307.svg
paper_authors.fullname: Abishek Sridhar
paper_authors.type: user
paper_authors.user: abisheksridhar
paper_authors._id: 64c1c37bec3c6181351f7492
paper_authors.name: Xianyi Cheng
paper_authors._id: 64c1c37bec3c6181351f7493
paper_authors.name: Yonatan Bisk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-27T10:48:56.774Z
paper_authors._id: 63180f3c879073cf83f135b0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63180f3c879073cf83f135b0/0ycXVKOywlU1hcvQ4MDws.jpeg
paper_authors.fullname: Yonatan Bisk
paper_authors.type: user
paper_authors.user: ybisk
paper_authors._id: 64c1c37bec3c6181351f7494
paper_authors.name: Daniel Fried
paper_authors._id: 64c1c37bec3c6181351f7495
paper_authors.name: Uri Alon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-27T10:49:37.442Z
paper_authors._id: 62cd8153248f9e6bc20ab250
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1657635637504-62cd8153248f9e6bc20ab250.jpeg
paper_authors.fullname: Uri Alon
paper_authors.type: user
paper_authors.user: urialon
paper_authors._id: 64c1c37bec3c6181351f7496
paper_authors.name: Graham Neubig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-27T10:50:34.852Z
paper_authors._id: 60de14638bedd2315529d43f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
paper_authors.fullname: Graham Neubig
paper_authors.type: user
paper_authors.user: gneubig
paper_title: WebArena: A Realistic Web Environment for Building Autonomous Agents
paper_summary: With generative AI advances, the exciting potential for autonomous agents to
  manage daily tasks via natural language commands has emerged. However, cur rent
  agents are primarily created and tested in simplified synthetic environments,
  substantially limiting real-world scenario representation. In this paper, we
  build an environment for agent command and control that is highly realistic and
  reproducible. Specifically, we focus on agents that perform tasks on websites,
  and we create an environment with fully functional websites from four common
  domains: e-commerce, social forum discussions, collaborative software
  development, and content management. Our environment is enriched with tools
  (e.g., a map) and external knowledge bases (e.g., user manuals) to encourage
  human-like task-solving. Building upon our environment, we release a set of
  benchmark tasks focusing on evaluating the functional correctness of task
  completions. The tasks in our benchmark are diverse, long-horizon, and are
  designed to emulate tasks that humans routinely perform on the internet. We
  design and implement several autonomous agents, integrating recent techniques
  such as reasoning before acting. The results demonstrate that solving complex
  tasks is challenging: our best GPT-4-based agent only achieves an end-to-end
  task success rate of 10.59%. These results highlight the need for further
  development of robust agents, that current state-of-the-art LMs are far from
  perfect performance in these real-life tasks, and that WebArena can be used to
  measure such progress. Our code, data, environment reproduction resources, and
  video demonstrations are publicly available at https://webarena.dev/.
paper_ai_summary: WebArena, a realistic and reproducible environment, evaluates the performance of autonomous agents performing complex tasks on websites using benchmark tasks that require human-like reasoning.
paper_ai_keywords: autonomous agents natural language commands synthetic environments realistic environment reproducible environment fully functional websites domains e-commerce social forum discussions collaborative software development content management tools external knowledge bases human-like task-solving benchmark tasks functional correctness end-to-end task success rate robust agents state-of-the-art LMs WebArena GPT-4