WebArena: A Realistic Web Environment for Building Autonomous Agents
With generative AI advances, the exciting potential for autonomous agents to
manage daily tasks via natural language commands has emerged. However, cur rent
agents are primarily created and tested in simplified synthetic environments,
substantially limiting real-world scenario representation. In this paper, we
build an environment for agent command and control that is highly realistic and
reproducible. Specifically, we focus on agents that perform tasks on websites,
and we create an environment with fully functional websites from four common
domains: e-commerce, social forum discussions, collaborative software
development, and content management. Our environment is enriched with tools
(e.g., a map) and external knowledge bases (e.g., user manuals) to encourage
human-like task-solving. Building upon our environment, we release a set of
benchmark tasks focusing on evaluating the functional correctness of task
completions. The tasks in our benchmark are diverse, long-horizon, and are
designed to emulate tasks that humans routinely perform on the internet. We
design and implement several autonomous agents, integrating recent techniques
such as reasoning before acting. The results demonstrate that solving complex
tasks is challenging: our best GPT-4-based agent only achieves an end-to-end
task success rate of 10.59%. These results highlight the need for further
development of robust agents, that current state-of-the-art LMs are far from
perfect performance in these real-life tasks, and that WebArena can be used to
measure such progress. Our code, data, environment reproduction resources, and
video demonstrations are publicly available at https://webarena.dev/.
64c1c37bec3c6181351f748c
Shuyan Zhou
64c1c37bec3c6181351f748d
Frank F. Xu
admin_assigned
2023-07-27T10:42:46.834Z
61ac0a0f57307cf7ae3a35bb
/avatars/c3b00cf275b2805861f5b020c9d00639.svg
Frank Xu
user
frankxu
64c1c37bec3c6181351f748e
Hao Zhu
claimed_verified
2024-03-14T12:31:41.475Z
61aa376688c20eebf1e8deb3
/avatars/7c11dcb232c73547d7d87834be287822.svg
Hao Zhu
user
ProKil
64c1c37bec3c6181351f748f
Xuhui Zhou
admin_assigned
2023-07-27T10:44:28.825Z
61873839b1085ab638324e84
/avatars/a02d8067cd3a1cc4d9acff70cbc84078.svg
Xuhui Zhou
user
Xuhui
64c1c37bec3c6181351f7490
Robert Lo
64c1c37bec3c6181351f7491
Abishek Sridhar
admin_assigned
2023-07-27T10:45:44.755Z
63bc400f141c7d395c4a9383
/avatars/3a494e3aa55096c506ebaecb3dee0307.svg
Abishek Sridhar
user
abisheksridhar
64c1c37bec3c6181351f7492
Xianyi Cheng
64c1c37bec3c6181351f7493
Yonatan Bisk
admin_assigned
2023-07-27T10:48:56.774Z
63180f3c879073cf83f135b0
https://cdn-avatars.huggingface.co/v1/production/uploads/63180f3c879073cf83f135b0/0ycXVKOywlU1hcvQ4MDws.jpeg
Yonatan Bisk
user
ybisk
64c1c37bec3c6181351f7494
Daniel Fried
64c1c37bec3c6181351f7495
Uri Alon
admin_assigned
2023-07-27T10:49:37.442Z
62cd8153248f9e6bc20ab250
https://cdn-avatars.huggingface.co/v1/production/uploads/1657635637504-62cd8153248f9e6bc20ab250.jpeg
Uri Alon
user
urialon
64c1c37bec3c6181351f7496
Graham Neubig
admin_assigned
2023-07-27T10:50:34.852Z
60de14638bedd2315529d43f
https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
Graham Neubig
user
gneubig
WebArena: A Realistic Web Environment for Building Autonomous Agents
With generative AI advances, the exciting potential for autonomous agents to
manage daily tasks via natural language commands has emerged. However, cur rent
agents are primarily created and tested in simplified synthetic environments,
substantially limiting real-world scenario representation. In this paper, we
build an environment for agent command and control that is highly realistic and
reproducible. Specifically, we focus on agents that perform tasks on websites,
and we create an environment with fully functional websites from four common
domains: e-commerce, social forum discussions, collaborative software
development, and content management. Our environment is enriched with tools
(e.g., a map) and external knowledge bases (e.g., user manuals) to encourage
human-like task-solving. Building upon our environment, we release a set of
benchmark tasks focusing on evaluating the functional correctness of task
completions. The tasks in our benchmark are diverse, long-horizon, and are
designed to emulate tasks that humans routinely perform on the internet. We
design and implement several autonomous agents, integrating recent techniques
such as reasoning before acting. The results demonstrate that solving complex
tasks is challenging: our best GPT-4-based agent only achieves an end-to-end
task success rate of 10.59%. These results highlight the need for further
development of robust agents, that current state-of-the-art LMs are far from
perfect performance in these real-life tasks, and that WebArena can be used to
measure such progress. Our code, data, environment reproduction resources, and
video demonstrations are publicly available at https://webarena.dev/.
WebArena, a realistic and reproducible environment, evaluates the performance of autonomous agents performing complex tasks on websites using benchmark tasks that require human-like reasoning.
autonomous agents natural language commands synthetic environments realistic environment reproducible environment fully functional websites domains e-commerce social forum discussions collaborative software development content management tools external knowledge bases human-like task-solving benchmark tasks functional correctness end-to-end task success rate robust agents state-of-the-art LMs WebArena GPT-4