The BrowserGym Ecosystem for Web Agent Research
The BrowserGym ecosystem addresses the growing need for efficient evaluation
and benchmarking of web agents, particularly those leveraging automation and
Large Language Models (LLMs) for web interaction tasks. Many existing
benchmarks suffer from fragmentation and inconsistent evaluation methodologies,
making it challenging to achieve reliable comparisons and reproducible results.
BrowserGym aims to solve this by providing a unified, gym-like environment with
well-defined observation and action spaces, facilitating standardized
evaluation across diverse benchmarks. Combined with AgentLab, a complementary
framework that aids in agent creation, testing, and analysis, BrowserGym offers
flexibility for integrating new benchmarks while ensuring consistent evaluation
and comprehensive experiment management. This standardized approach seeks to
reduce the time and complexity of developing web agents, supporting more
reliable comparisons and facilitating in-depth analysis of agent behaviors, and
could result in more adaptable, capable agents, ultimately accelerating
innovation in LLM-driven automation. As a supporting evidence, we conduct the
first large-scale, multi-benchmark web agent experiment and compare the
performance of 6 state-of-the-art LLMs across all benchmarks currently
available in BrowserGym. Among other findings, our results highlight a large
discrepancy between OpenAI and Anthropic's latests models, with
Claude-3.5-Sonnet leading the way on almost all benchmarks, except on
vision-related tasks where GPT-4o is superior. Despite these advancements, our
results emphasize that building robust and efficient web agents remains a
significant challenge, due to the inherent complexity of real-world web
environments and the limitations of current models.
675b2a535b0522bba0448eaa
Thibault Le Sellier De Chezelles
claimed_verified
2024-12-13T15:23:00.318Z
633e78d226bad5c614ff9acf
/avatars/cfbc94055207ed7c3e9a1065f8bd96c8.svg
ThibaultLSDC
user
ThibaultLSDC
675b2a535b0522bba0448eab
Maxime Gasse
claimed_verified
2024-12-13T15:23:02.118Z
6442a3e53610a28a4ad49075
/avatars/bf2149a2520d89ecd379aff05cdd8eeb.svg
Maxime Gasse
user
gasse
675b2a535b0522bba0448eac
Alexandre Drouin
675b2a535b0522bba0448ead
Massimo Caccia
claimed_verified
2024-12-13T15:22:55.145Z
64b04320e5000ae8a57570a7
/avatars/226f57ea23b9dd861e71b7ba7821adce.svg
Massimo Caccia
user
optimass
675b2a535b0522bba0448eae
Léo Boisvert
claimed_verified
2024-12-13T15:22:57.465Z
6671e06d6d27e245f625b7e2
https://cdn-avatars.huggingface.co/v1/production/uploads/6671e06d6d27e245f625b7e2/alb0_Tlo8IOBDShWwMQrs.png
Léo Boisvert
user
jardinet-souffleton
675b2a535b0522bba0448eaf
Megh Thakkar
675b2a535b0522bba0448eb0
Tom Marty
675b2a535b0522bba0448eb1
Rim Assouel
675b2a535b0522bba0448eb2
Sahar Omidi Shayegan
675b2a535b0522bba0448eb3
Lawrence Keunho Jang
claimed_verified
2024-12-20T08:33:49.470Z
664aebe829eadb3ab4e4ca3f
/avatars/548d5656e082c8959ac78b883f0805af.svg
Lawrence Jang
user
ljang0
675b2a535b0522bba0448eb4
Xing Han Lù
claimed_verified
2024-12-13T15:23:03.754Z
5fa9ff3ea13e063b8b2b60cb
https://cdn-avatars.huggingface.co/v1/production/uploads/1633380224986-5fa9ff3ea13e063b8b2b60cb.jpeg
Xing Han Lù
user
xhluca
675b2a535b0522bba0448eb5
Ori Yoran
675b2a535b0522bba0448eb6
Dehan Kong
675b2a535b0522bba0448eb7
Frank F. Xu
675b2a535b0522bba0448eb8
Siva Reddy
675b2a535b0522bba0448eb9
Quentin Cappart
675b2a535b0522bba0448eba
Graham Neubig
675b2a535b0522bba0448ebb
Ruslan Salakhutdinov
675b2a535b0522bba0448ebc
Nicolas Chapados
675b2a535b0522bba0448ebd
Alexandre Lacoste
The BrowserGym Ecosystem for Web Agent Research
The BrowserGym ecosystem addresses the growing need for efficient evaluation
and benchmarking of web agents, particularly those leveraging automation and
Large Language Models (LLMs) for web interaction tasks. Many existing
benchmarks suffer from fragmentation and inconsistent evaluation methodologies,
making it challenging to achieve reliable comparisons and reproducible results.
BrowserGym aims to solve this by providing a unified, gym-like environment with
well-defined observation and action spaces, facilitating standardized
evaluation across diverse benchmarks. Combined with AgentLab, a complementary
framework that aids in agent creation, testing, and analysis, BrowserGym offers
flexibility for integrating new benchmarks while ensuring consistent evaluation
and comprehensive experiment management. This standardized approach seeks to
reduce the time and complexity of developing web agents, supporting more
reliable comparisons and facilitating in-depth analysis of agent behaviors, and
could result in more adaptable, capable agents, ultimately accelerating
innovation in LLM-driven automation. As a supporting evidence, we conduct the
first large-scale, multi-benchmark web agent experiment and compare the
performance of 6 state-of-the-art LLMs across all benchmarks currently
available in BrowserGym. Among other findings, our results highlight a large
discrepancy between OpenAI and Anthropic's latests models, with
Claude-3.5-Sonnet leading the way on almost all benchmarks, except on
vision-related tasks where GPT-4o is superior. Despite these advancements, our
results emphasize that building robust and efficient web agents remains a
significant challenge, due to the inherent complexity of real-world web
environments and the limitations of current models.
BrowserGym provides a standardized environment for evaluating web agents using LLMs, facilitating consistent benchmarking and improving agent development and analysis.
Large Language Models gym-like environment observation and action spaces AgentLab web agents multi-benchmark evaluation Claude-3.5-Sonnet GPT-4o real-world web environments