A Real-World WebAgent with Planning, Long Context Understanding, and
  Program Synthesis
Pre-trained large language models (LLMs) have recently achieved better
generalization and sample efficiency in autonomous web navigation. However, the
performance on real-world websites has still suffered from (1) open domainness,
(2) limited context length, and (3) lack of inductive bias on HTML. We
introduce WebAgent, an LLM-driven agent that can complete the tasks on real
websites following natural language instructions. WebAgent plans ahead by
decomposing instructions into canonical sub-instructions, summarizes long HTML
documents into task-relevant snippets, and acts on websites via generated
Python programs from those. We design WebAgent with Flan-U-PaLM, for grounded
code generation, and HTML-T5, new pre-trained LLMs for long HTML documents
using local and global attention mechanisms and a mixture of long-span
denoising objectives, for planning and summarization. We empirically
demonstrate that our recipe improves the success on a real website by over 50%,
and that HTML-T5 is the best model to solve HTML-based tasks; achieving 14.9%
higher success rate than prior SoTA on the MiniWoB web navigation benchmark and
better accuracy on offline task planning evaluation.
64bf412046cc3cdfbb29e894
Izzeddin Gur
claimed_verified
2023-07-26T08:44:06.313Z
64c0ccc19832a455bc2a2069
/avatars/9d4fdf13cd62c6c258b5f50d67c8a1b1.svg
IZZEDDIN GUR
user
izzeddingur
64bf412046cc3cdfbb29e895
Hiroki Furuta
admin_assigned
2023-07-25T08:14:29.475Z
646c39fc3e2a7b06594bac6c
/avatars/8925fef2b6e7105a0f0a4bbf160cfda3.svg
Hiroki Furuta
user
frt03
64bf412046cc3cdfbb29e896
Austin Huang
admin_assigned
2023-07-25T08:15:21.796Z
6020b14c8345068224043bc0
/avatars/1c5c02f9165e188ab81a53efabc062e2.svg
Austin Huang
user
austinvhuang
64bf412046cc3cdfbb29e897
Mustafa Safdari
admin_assigned
2023-07-25T08:15:49.843Z
6430b08b74e7df5793edf353
/avatars/67875a201ffc22416acbf74b759b544b.svg
Mustafa Safdari
user
TheMegusAlexandrus
64bf412046cc3cdfbb29e898
Yutaka Matsuo
64bf412046cc3cdfbb29e899
Douglas Eck
64bf412046cc3cdfbb29e89a
Aleksandra Faust
admin_assigned
2023-07-25T08:17:22.594Z
64a3b828225b2121f6b3beb3
https://cdn-avatars.huggingface.co/v1/production/uploads/64a3b828225b2121f6b3beb3/44AgAQENc7cJMOWcoK4g_.jpeg
Aleksandra Faust
user
sandraorion
A Real-World WebAgent with Planning, Long Context Understanding, and
  Program Synthesis
Pre-trained large language models (LLMs) have recently achieved better
generalization and sample efficiency in autonomous web navigation. However, the
performance on real-world websites has still suffered from (1) open domainness,
(2) limited context length, and (3) lack of inductive bias on HTML. We
introduce WebAgent, an LLM-driven agent that can complete the tasks on real
websites following natural language instructions. WebAgent plans ahead by
decomposing instructions into canonical sub-instructions, summarizes long HTML
documents into task-relevant snippets, and acts on websites via generated
Python programs from those. We design WebAgent with Flan-U-PaLM, for grounded
code generation, and HTML-T5, new pre-trained LLMs for long HTML documents
using local and global attention mechanisms and a mixture of long-span
denoising objectives, for planning and summarization. We empirically
demonstrate that our recipe improves the success on a real website by over 50%,
and that HTML-T5 is the best model to solve HTML-based tasks; achieving 14.9%
higher success rate than prior SoTA on the MiniWoB web navigation benchmark and
better accuracy on offline task planning evaluation.
WebAgent uses Flan-U-PaLM and HTML-T5 to improve task success in real-world web navigation through decomposed instructions, HTML summarization, and Python code generation.
pre-trained large language models LLMs autonomous web navigation open domainness context length inductive bias HTML WebAgent canonical sub-instructions task-relevant snippets Python programs grounded code generation local attention mechanisms global attention mechanisms long-span denoising objectives MiniWoB web navigation benchmark task planning evaluation