title: The BrowserGym Ecosystem for Web Agent Research
summary: The BrowserGym ecosystem addresses the growing need for efficient evaluation
  and benchmarking of web agents, particularly those leveraging automation and
  Large Language Models (LLMs) for web interaction tasks. Many existing
  benchmarks suffer from fragmentation and inconsistent evaluation methodologies,
  making it challenging to achieve reliable comparisons and reproducible results.
  BrowserGym aims to solve this by providing a unified, gym-like environment with
  well-defined observation and action spaces, facilitating standardized
  evaluation across diverse benchmarks. Combined with AgentLab, a complementary
  framework that aids in agent creation, testing, and analysis, BrowserGym offers
  flexibility for integrating new benchmarks while ensuring consistent evaluation
  and comprehensive experiment management. This standardized approach seeks to
  reduce the time and complexity of developing web agents, supporting more
  reliable comparisons and facilitating in-depth analysis of agent behaviors, and
  could result in more adaptable, capable agents, ultimately accelerating
  innovation in LLM-driven automation. As a supporting evidence, we conduct the
  first large-scale, multi-benchmark web agent experiment and compare the
  performance of 6 state-of-the-art LLMs across all benchmarks currently
  available in BrowserGym. Among other findings, our results highlight a large
  discrepancy between OpenAI and Anthropic's latests models, with
  Claude-3.5-Sonnet leading the way on almost all benchmarks, except on
  vision-related tasks where GPT-4o is superior. Despite these advancements, our
  results emphasize that building robust and efficient web agents remains a
  significant challenge, due to the inherent complexity of real-world web
  environments and the limitations of current models.
paper_authors._id: 675b2a535b0522bba0448eaa
paper_authors.name: Thibault Le Sellier De Chezelles
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:23:00.318Z
paper_authors._id: 633e78d226bad5c614ff9acf
paper_authors.avatarUrl: /avatars/cfbc94055207ed7c3e9a1065f8bd96c8.svg
paper_authors.fullname: ThibaultLSDC
paper_authors.type: user
paper_authors.user: ThibaultLSDC
paper_authors._id: 675b2a535b0522bba0448eab
paper_authors.name: Maxime Gasse
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:23:02.118Z
paper_authors._id: 6442a3e53610a28a4ad49075
paper_authors.avatarUrl: /avatars/bf2149a2520d89ecd379aff05cdd8eeb.svg
paper_authors.fullname: Maxime Gasse
paper_authors.type: user
paper_authors.user: gasse
paper_authors._id: 675b2a535b0522bba0448eac
paper_authors.name: Alexandre Drouin
paper_authors._id: 675b2a535b0522bba0448ead
paper_authors.name: Massimo Caccia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:22:55.145Z
paper_authors._id: 64b04320e5000ae8a57570a7
paper_authors.avatarUrl: /avatars/226f57ea23b9dd861e71b7ba7821adce.svg
paper_authors.fullname: Massimo Caccia
paper_authors.type: user
paper_authors.user: optimass
paper_authors._id: 675b2a535b0522bba0448eae
paper_authors.name: Léo Boisvert
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:22:57.465Z
paper_authors._id: 6671e06d6d27e245f625b7e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6671e06d6d27e245f625b7e2/alb0_Tlo8IOBDShWwMQrs.png
paper_authors.fullname: Léo Boisvert
paper_authors.type: user
paper_authors.user: jardinet-souffleton
paper_authors._id: 675b2a535b0522bba0448eaf
paper_authors.name: Megh Thakkar
paper_authors._id: 675b2a535b0522bba0448eb0
paper_authors.name: Tom Marty
paper_authors._id: 675b2a535b0522bba0448eb1
paper_authors.name: Rim Assouel
paper_authors._id: 675b2a535b0522bba0448eb2
paper_authors.name: Sahar Omidi Shayegan
paper_authors._id: 675b2a535b0522bba0448eb3
paper_authors.name: Lawrence Keunho Jang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:33:49.470Z
paper_authors._id: 664aebe829eadb3ab4e4ca3f
paper_authors.avatarUrl: /avatars/548d5656e082c8959ac78b883f0805af.svg
paper_authors.fullname: Lawrence Jang
paper_authors.type: user
paper_authors.user: ljang0
paper_authors._id: 675b2a535b0522bba0448eb4
paper_authors.name: Xing Han Lù
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:23:03.754Z
paper_authors._id: 5fa9ff3ea13e063b8b2b60cb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1633380224986-5fa9ff3ea13e063b8b2b60cb.jpeg
paper_authors.fullname: Xing Han Lù
paper_authors.type: user
paper_authors.user: xhluca
paper_authors._id: 675b2a535b0522bba0448eb5
paper_authors.name: Ori Yoran
paper_authors._id: 675b2a535b0522bba0448eb6
paper_authors.name: Dehan Kong
paper_authors._id: 675b2a535b0522bba0448eb7
paper_authors.name: Frank F. Xu
paper_authors._id: 675b2a535b0522bba0448eb8
paper_authors.name: Siva Reddy
paper_authors._id: 675b2a535b0522bba0448eb9
paper_authors.name: Quentin Cappart
paper_authors._id: 675b2a535b0522bba0448eba
paper_authors.name: Graham Neubig
paper_authors._id: 675b2a535b0522bba0448ebb
paper_authors.name: Ruslan Salakhutdinov
paper_authors._id: 675b2a535b0522bba0448ebc
paper_authors.name: Nicolas Chapados
paper_authors._id: 675b2a535b0522bba0448ebd
paper_authors.name: Alexandre Lacoste
paper_title: The BrowserGym Ecosystem for Web Agent Research
paper_summary: The BrowserGym ecosystem addresses the growing need for efficient evaluation
  and benchmarking of web agents, particularly those leveraging automation and
  Large Language Models (LLMs) for web interaction tasks. Many existing
  benchmarks suffer from fragmentation and inconsistent evaluation methodologies,
  making it challenging to achieve reliable comparisons and reproducible results.
  BrowserGym aims to solve this by providing a unified, gym-like environment with
  well-defined observation and action spaces, facilitating standardized
  evaluation across diverse benchmarks. Combined with AgentLab, a complementary
  framework that aids in agent creation, testing, and analysis, BrowserGym offers
  flexibility for integrating new benchmarks while ensuring consistent evaluation
  and comprehensive experiment management. This standardized approach seeks to
  reduce the time and complexity of developing web agents, supporting more
  reliable comparisons and facilitating in-depth analysis of agent behaviors, and
  could result in more adaptable, capable agents, ultimately accelerating
  innovation in LLM-driven automation. As a supporting evidence, we conduct the
  first large-scale, multi-benchmark web agent experiment and compare the
  performance of 6 state-of-the-art LLMs across all benchmarks currently
  available in BrowserGym. Among other findings, our results highlight a large
  discrepancy between OpenAI and Anthropic's latests models, with
  Claude-3.5-Sonnet leading the way on almost all benchmarks, except on
  vision-related tasks where GPT-4o is superior. Despite these advancements, our
  results emphasize that building robust and efficient web agents remains a
  significant challenge, due to the inherent complexity of real-world web
  environments and the limitations of current models.
paper_ai_summary: BrowserGym provides a standardized environment for evaluating web agents using LLMs, facilitating consistent benchmarking and improving agent development and analysis.
paper_ai_keywords: Large Language Models gym-like environment observation and action spaces AgentLab web agents multi-benchmark evaluation Claude-3.5-Sonnet GPT-4o real-world web environments