title: WebVoyager: Building an End-to-End Web Agent with Large Multimodal
    Models
summary: The advancement of large language models (LLMs) leads to a new era marked by
  the development of autonomous applications in the real world, which drives
  innovation in the creation of advanced web-based agents. Existing web agents
  typically only handle one input modality and are evaluated only in simplified
  web simulators or static web snapshots, greatly limiting their applicability in
  real-world scenarios. To bridge this gap, we introduce WebVoyager, an
  innovative Large Multimodal Model (LMM) powered web agent that can complete
  user instructions end-to-end by interacting with real-world websites. Moreover,
  we propose a new evaluation protocol for web agents to address the challenges
  of automatic evaluation of open-ended web agent tasks, leveraging the robust
  multimodal comprehension capabilities of GPT-4V. We create a new benchmark by
  gathering real-world tasks from 15 widely used websites to evaluate our agents.
  We show that WebVoyager achieves a 55.7% task success rate, significantly
  surpassing the performance of both GPT-4 (All Tools) and the WebVoyager
  (text-only) setups, underscoring the exceptional capability of WebVoyager in
  practical applications. We found that our proposed automatic evaluation
  achieves 85.3% agreement with human judgment, paving the way for further
  development of web agents in a real-world setting.
paper_authors._id: 65b318d4dfa3789ef34877c4
paper_authors.name: Hongliang He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T09:58:34.239Z
paper_authors._id: 64e863d59c7e43f8a056a332
paper_authors.avatarUrl: /avatars/44e411f06b70a9a030a452b8d5f8e663.svg
paper_authors.fullname: Hongliang He
paper_authors.type: user
paper_authors.user: Hongliang1997
paper_authors._id: 65b318d4dfa3789ef34877c5
paper_authors.name: Wenlin Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T09:58:41.216Z
paper_authors._id: 634f18e4aae4bde2c8e2adca
paper_authors.avatarUrl: /avatars/40549a59fc5ba04a4baa5a1d5dba0847.svg
paper_authors.fullname: Wenlin Yao
paper_authors.type: user
paper_authors.user: wenlinyao
paper_authors._id: 65b318d4dfa3789ef34877c6
paper_authors.name: Kaixin Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T09:58:52.341Z
paper_authors._id: 64ae4f6280f308a395fd7c19
paper_authors.avatarUrl: /avatars/5f1330f8187cd5e66aa517303659f110.svg
paper_authors.fullname: Kaixin Ma
paper_authors.type: user
paper_authors.user: kaixinm
paper_authors._id: 65b318d4dfa3789ef34877c7
paper_authors.name: Wenhao Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T09:59:25.043Z
paper_authors._id: 5feab3a28a3201f8e554c969
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660795228685-5feab3a28a3201f8e554c969.png
paper_authors.fullname: Wenhao Yu
paper_authors.type: user
paper_authors.user: wyu1
paper_authors._id: 65b318d4dfa3789ef34877c8
paper_authors.name: Yong Dai
paper_authors._id: 65b318d4dfa3789ef34877c9
paper_authors.name: Hongming Zhang
paper_authors._id: 65b318d4dfa3789ef34877ca
paper_authors.name: Zhenzhong Lan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-26T10:00:19.736Z
paper_authors._id: 62ce6dd785cfd21c04c7e6f5
paper_authors.avatarUrl: /avatars/89837a5dea6e2d753d59caad142bed4a.svg
paper_authors.fullname: ZhenzhongLan
paper_authors.type: user
paper_authors.user: DannyLan
paper_authors._id: 65b318d4dfa3789ef34877cb
paper_authors.name: Dong Yu
paper_title: WebVoyager: Building an End-to-End Web Agent with Large Multimodal
    Models
paper_summary: The advancement of large language models (LLMs) leads to a new era marked by
  the development of autonomous applications in the real world, which drives
  innovation in the creation of advanced web-based agents. Existing web agents
  typically only handle one input modality and are evaluated only in simplified
  web simulators or static web snapshots, greatly limiting their applicability in
  real-world scenarios. To bridge this gap, we introduce WebVoyager, an
  innovative Large Multimodal Model (LMM) powered web agent that can complete
  user instructions end-to-end by interacting with real-world websites. Moreover,
  we propose a new evaluation protocol for web agents to address the challenges
  of automatic evaluation of open-ended web agent tasks, leveraging the robust
  multimodal comprehension capabilities of GPT-4V. We create a new benchmark by
  gathering real-world tasks from 15 widely used websites to evaluate our agents.
  We show that WebVoyager achieves a 55.7% task success rate, significantly
  surpassing the performance of both GPT-4 (All Tools) and the WebVoyager
  (text-only) setups, underscoring the exceptional capability of WebVoyager in
  practical applications. We found that our proposed automatic evaluation
  achieves 85.3% agreement with human judgment, paving the way for further
  development of web agents in a real-world setting.
paper_ai_keywords: Large Multimodal Model (LMM) webVoyager multimodal comprehension GPT-4V task success rate automatic evaluation human judgment