WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
We propose the problem of conversational web navigation, where a digital
agent controls a web browser and follows user instructions to solve real-world
tasks in a multi-turn dialogue fashion. To support this problem, we introduce
WEBLINX - a large-scale benchmark of 100K interactions across 2300 expert
demonstrations of conversational web navigation. Our benchmark covers a broad
range of patterns on over 150 real-world websites and can be used to train and
evaluate agents in diverse scenarios. Due to the magnitude of information
present, Large Language Models (LLMs) cannot process entire web pages in
real-time. To solve this bottleneck, we design a retrieval-inspired model that
efficiently prunes HTML pages by ranking relevant elements. We use the selected
elements, along with screenshots and action history, to assess a variety of
models for their ability to replicate human behavior when navigating the web.
Our experiments span from small text-only to proprietary multimodal LLMs. We
find that smaller finetuned decoders surpass the best zero-shot LLMs (including
GPT-4V), but also larger finetuned multimodal models which were explicitly
pretrained on screenshots. However, all finetuned models struggle to generalize
to unseen websites. Our findings highlight the need for large multimodal models
that can generalize to novel settings. Our code, data and models are available
for research: https://mcgill-nlp.github.io/weblinx
65c59241ed68e7aafdc5cace
Xing Han Lù
claimed_verified
2024-02-09T08:05:07.030Z
5fa9ff3ea13e063b8b2b60cb
https://cdn-avatars.huggingface.co/v1/production/uploads/1633380224986-5fa9ff3ea13e063b8b2b60cb.jpeg
Xing Han Lù
user
xhluca
65c59241ed68e7aafdc5cacf
Zdeněk Kasner
admin_assigned
2024-02-09T10:18:47.386Z
631b8456bf1351ed2bd1d332
https://cdn-avatars.huggingface.co/v1/production/uploads/1662747897741-631b8456bf1351ed2bd1d332.jpeg
Zdeněk Kasner
user
kasnerz
65c59241ed68e7aafdc5cad0
Siva Reddy
extracted_confirmed
2025-04-11T03:27:01.619Z
624734dc4c731bb6bfab8af7
/avatars/6b250b58710a3287b85e4733c1824558.svg
Siva Reddy
user
sivareddyg
WebLINX: Real-World Website Navigation with Multi-Turn Dialogue
We propose the problem of conversational web navigation, where a digital
agent controls a web browser and follows user instructions to solve real-world
tasks in a multi-turn dialogue fashion. To support this problem, we introduce
WEBLINX - a large-scale benchmark of 100K interactions across 2300 expert
demonstrations of conversational web navigation. Our benchmark covers a broad
range of patterns on over 150 real-world websites and can be used to train and
evaluate agents in diverse scenarios. Due to the magnitude of information
present, Large Language Models (LLMs) cannot process entire web pages in
real-time. To solve this bottleneck, we design a retrieval-inspired model that
efficiently prunes HTML pages by ranking relevant elements. We use the selected
elements, along with screenshots and action history, to assess a variety of
models for their ability to replicate human behavior when navigating the web.
Our experiments span from small text-only to proprietary multimodal LLMs. We
find that smaller finetuned decoders surpass the best zero-shot LLMs (including
GPT-4V), but also larger finetuned multimodal models which were explicitly
pretrained on screenshots. However, all finetuned models struggle to generalize
to unseen websites. Our findings highlight the need for large multimodal models
that can generalize to novel settings. Our code, data and models are available
for research: https://mcgill-nlp.github.io/weblinx
A benchmark and retrieval-inspired model for efficient conversational web navigation by digital agents across diverse real-world websites is introduced, highlighting challenges in generalization to unseen sites.
conversational web navigation WEBLINX benchmark LLMs large language models HTML pages retrieval-inspired model human behavior replication multimodal LLMs zero-shot LLMs GPT-4V text-only models proprietary multimodal LLMs generalization