title: AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
summary: Language agents, built on top of language models (LMs), are systems that can
  interact with complex environments, such as the open web. In this work, we
  examine whether such agents can perform realistic and time-consuming tasks on
  the web, e.g., monitoring real-estate markets or locating relevant nearby
  businesses. We introduce AssistantBench, a challenging new benchmark consisting
  of 214 realistic tasks that can be automatically evaluated, covering different
  scenarios and domains. We find that AssistantBench exposes the limitations of
  current systems, including language models and retrieval-augmented language
  models, as no model reaches an accuracy of more than 25 points. While
  closed-book LMs perform well, they exhibit low precision since they tend to
  hallucinate facts. State-of-the-art web agents reach a score of near zero.
  Additionally, we introduce SeePlanAct (SPA), a new web agent that significantly
  outperforms previous agents, and an ensemble of SPA and closed-book models
  reaches the best overall performance. Moreover, we analyze failures of current
  systems and highlight that web navigation remains a major challenge.
paper_authors._id: 669f26d53de1a03833d0db5e
paper_authors.name: Ori Yoran
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-23T07:47:23.933Z
paper_authors._id: 610b7ccff5dd7bcefedaff57
paper_authors.avatarUrl: /avatars/a1e77460105a56f84371fa02d530a71e.svg
paper_authors.fullname: Yoran
paper_authors.type: user
paper_authors.user: Ori
paper_authors._id: 669f26d53de1a03833d0db5f
paper_authors.name: Samuel Joseph Amouyal
paper_authors._id: 669f26d53de1a03833d0db60
paper_authors.name: Chaitanya Malaviya
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-23T08:01:24.934Z
paper_authors._id: 646bad133e2a7b065948d4be
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646bad133e2a7b065948d4be/Hk21DxumaBbkMJNZCKL4A.jpeg
paper_authors.fullname: Chaitanya Malaviya
paper_authors.type: user
paper_authors.user: cmalaviya
paper_authors._id: 669f26d53de1a03833d0db61
paper_authors.name: Ben Bogin
paper_authors._id: 669f26d53de1a03833d0db62
paper_authors.name: Ofir Press
paper_authors._id: 669f26d53de1a03833d0db63
paper_authors.name: Jonathan Berant
paper_title: AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?
paper_summary: Language agents, built on top of language models (LMs), are systems that can
  interact with complex environments, such as the open web. In this work, we
  examine whether such agents can perform realistic and time-consuming tasks on
  the web, e.g., monitoring real-estate markets or locating relevant nearby
  businesses. We introduce AssistantBench, a challenging new benchmark consisting
  of 214 realistic tasks that can be automatically evaluated, covering different
  scenarios and domains. We find that AssistantBench exposes the limitations of
  current systems, including language models and retrieval-augmented language
  models, as no model reaches an accuracy of more than 25 points. While
  closed-book LMs perform well, they exhibit low precision since they tend to
  hallucinate facts. State-of-the-art web agents reach a score of near zero.
  Additionally, we introduce SeePlanAct (SPA), a new web agent that significantly
  outperforms previous agents, and an ensemble of SPA and closed-book models
  reaches the best overall performance. Moreover, we analyze failures of current
  systems and highlight that web navigation remains a major challenge.
paper_ai_summary: AssistantBench evaluates web agents on realistic tasks, revealing limitations of current systems, with SeePlanAct achieving the best performance and highlighting web navigation as a key challenge.
paper_ai_keywords: language models language agents web agents retrieval-augmented language models AssistantBench SeePlanAct web navigation