title: WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
summary: We propose the problem of conversational web navigation, where a digital
  agent controls a web browser and follows user instructions to solve real-world
  tasks in a multi-turn dialogue fashion. To support this problem, we introduce
  WEBLINX - a large-scale benchmark of 100K interactions across 2300 expert
  demonstrations of conversational web navigation. Our benchmark covers a broad
  range of patterns on over 150 real-world websites and can be used to train and
  evaluate agents in diverse scenarios. Due to the magnitude of information
  present, Large Language Models (LLMs) cannot process entire web pages in
  real-time. To solve this bottleneck, we design a retrieval-inspired model that
  efficiently prunes HTML pages by ranking relevant elements. We use the selected
  elements, along with screenshots and action history, to assess a variety of
  models for their ability to replicate human behavior when navigating the web.
  Our experiments span from small text-only to proprietary multimodal LLMs. We
  find that smaller finetuned decoders surpass the best zero-shot LLMs (including
  GPT-4V), but also larger finetuned multimodal models which were explicitly
  pretrained on screenshots. However, all finetuned models struggle to generalize
  to unseen websites. Our findings highlight the need for large multimodal models
  that can generalize to novel settings. Our code, data and models are available
  for research: https://mcgill-nlp.github.io/weblinx
paper_authors._id: 65c59241ed68e7aafdc5cace
paper_authors.name: Xing Han Lù
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-09T08:05:07.030Z
paper_authors._id: 5fa9ff3ea13e063b8b2b60cb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1633380224986-5fa9ff3ea13e063b8b2b60cb.jpeg
paper_authors.fullname: Xing Han Lù
paper_authors.type: user
paper_authors.user: xhluca
paper_authors._id: 65c59241ed68e7aafdc5cacf
paper_authors.name: Zdeněk Kasner
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:18:47.386Z
paper_authors._id: 631b8456bf1351ed2bd1d332
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662747897741-631b8456bf1351ed2bd1d332.jpeg
paper_authors.fullname: Zdeněk Kasner
paper_authors.type: user
paper_authors.user: kasnerz
paper_authors._id: 65c59241ed68e7aafdc5cad0
paper_authors.name: Siva Reddy
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-04-11T03:27:01.619Z
paper_authors._id: 624734dc4c731bb6bfab8af7
paper_authors.avatarUrl: /avatars/6b250b58710a3287b85e4733c1824558.svg
paper_authors.fullname: Siva Reddy
paper_authors.type: user
paper_authors.user: sivareddyg
paper_title: WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
paper_summary: We propose the problem of conversational web navigation, where a digital
  agent controls a web browser and follows user instructions to solve real-world
  tasks in a multi-turn dialogue fashion. To support this problem, we introduce
  WEBLINX - a large-scale benchmark of 100K interactions across 2300 expert
  demonstrations of conversational web navigation. Our benchmark covers a broad
  range of patterns on over 150 real-world websites and can be used to train and
  evaluate agents in diverse scenarios. Due to the magnitude of information
  present, Large Language Models (LLMs) cannot process entire web pages in
  real-time. To solve this bottleneck, we design a retrieval-inspired model that
  efficiently prunes HTML pages by ranking relevant elements. We use the selected
  elements, along with screenshots and action history, to assess a variety of
  models for their ability to replicate human behavior when navigating the web.
  Our experiments span from small text-only to proprietary multimodal LLMs. We
  find that smaller finetuned decoders surpass the best zero-shot LLMs (including
  GPT-4V), but also larger finetuned multimodal models which were explicitly
  pretrained on screenshots. However, all finetuned models struggle to generalize
  to unseen websites. Our findings highlight the need for large multimodal models
  that can generalize to novel settings. Our code, data and models are available
  for research: https://mcgill-nlp.github.io/weblinx
paper_ai_summary: A benchmark and retrieval-inspired model for efficient conversational web navigation by digital agents across diverse real-world websites is introduced, highlighting challenges in generalization to unseen sites.
paper_ai_keywords: conversational web navigation WEBLINX benchmark LLMs large language models HTML pages retrieval-inspired model human behavior replication multimodal LLMs zero-shot LLMs GPT-4V text-only models proprietary multimodal LLMs generalization