Multimodal Web Navigation with Instruction-Finetuned Foundation Models
The progress of autonomous web navigation has been hindered by the dependence
on billions of exploratory interactions via online reinforcement learning, and
domain-specific model designs that make it difficult to leverage generalization
from rich out-of-domain data. In this work, we study data-driven offline
training for web agents with vision-language foundation models. We propose an
instruction-following multimodal agent, WebGUM, that observes both webpage
screenshots and HTML pages and outputs web navigation actions, such as click
and type. WebGUM is trained by jointly finetuning an instruction-finetuned
language model and a vision transformer on a large corpus of demonstrations. We
empirically demonstrate this recipe improves the agent's ability of grounded
visual perception, HTML comprehension and multi-step reasoning, outperforming
prior works by a significant margin. On the MiniWoB benchmark, we improve over
the previous best offline methods by more than 31.9%, being close to reaching
online-finetuned SoTA. On the WebShop benchmark, our 3-billion-parameter model
achieves superior performance to the existing SoTA, PaLM-540B. We also collect
347K high-quality demonstrations using our trained models, 38 times larger than
prior work, and make them available to promote future research in this
direction.
646af37ac37ca1e1230d3504
Hiroki Furuta
claimed_verified
2023-05-23T06:39:29.879Z
646c39fc3e2a7b06594bac6c
/avatars/8925fef2b6e7105a0f0a4bbf160cfda3.svg
Hiroki Furuta
user
frt03
646af37ac37ca1e1230d3505
Ofir Nachum
646af37ac37ca1e1230d3506
Kuang-Huei Lee
646af37ac37ca1e1230d3507
Yutaka Matsuo
646af37ac37ca1e1230d3508
Shixiang Shane Gu
admin_assigned
2023-05-26T08:52:31.752Z
631ea400e207d8fe9568dcb2
/avatars/ab972dc03fcb67ea76378c6b8f7a2622.svg
Shane Gu
user
shanegu
646af37ac37ca1e1230d3509
Izzeddin Gur
Multimodal Web Navigation with Instruction-Finetuned Foundation Models
The progress of autonomous web navigation has been hindered by the dependence
on billions of exploratory interactions via online reinforcement learning, and
domain-specific model designs that make it difficult to leverage generalization
from rich out-of-domain data. In this work, we study data-driven offline
training for web agents with vision-language foundation models. We propose an
instruction-following multimodal agent, WebGUM, that observes both webpage
screenshots and HTML pages and outputs web navigation actions, such as click
and type. WebGUM is trained by jointly finetuning an instruction-finetuned
language model and a vision transformer on a large corpus of demonstrations. We
empirically demonstrate this recipe improves the agent's ability of grounded
visual perception, HTML comprehension and multi-step reasoning, outperforming
prior works by a significant margin. On the MiniWoB benchmark, we improve over
the previous best offline methods by more than 31.9%, being close to reaching
online-finetuned SoTA. On the WebShop benchmark, our 3-billion-parameter model
achieves superior performance to the existing SoTA, PaLM-540B. We also collect
347K high-quality demonstrations using our trained models, 38 times larger than
prior work, and make them available to promote future research in this
direction.
An instruction-following multimodal agent named WebGUM, trained offline using vision transformers and language models, significantly outperforms existing methods in web navigation and multi-step reasoning tasks.
offline training web agents vision-language foundation models instruction-following vision transformer multimodal agent grounded visual perception HTML comprehension multi-step reasoning MiniWoB benchmark WebShop benchmark PaLM-540B high-quality demonstrations