title: Multimodal Web Navigation with Instruction-Finetuned Foundation Models
summary: The progress of autonomous web navigation has been hindered by the dependence
  on billions of exploratory interactions via online reinforcement learning, and
  domain-specific model designs that make it difficult to leverage generalization
  from rich out-of-domain data. In this work, we study data-driven offline
  training for web agents with vision-language foundation models. We propose an
  instruction-following multimodal agent, WebGUM, that observes both webpage
  screenshots and HTML pages and outputs web navigation actions, such as click
  and type. WebGUM is trained by jointly finetuning an instruction-finetuned
  language model and a vision transformer on a large corpus of demonstrations. We
  empirically demonstrate this recipe improves the agent's ability of grounded
  visual perception, HTML comprehension and multi-step reasoning, outperforming
  prior works by a significant margin. On the MiniWoB benchmark, we improve over
  the previous best offline methods by more than 31.9%, being close to reaching
  online-finetuned SoTA. On the WebShop benchmark, our 3-billion-parameter model
  achieves superior performance to the existing SoTA, PaLM-540B. We also collect
  347K high-quality demonstrations using our trained models, 38 times larger than
  prior work, and make them available to promote future research in this
  direction.
paper_authors._id: 646af37ac37ca1e1230d3504
paper_authors.name: Hiroki Furuta
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-23T06:39:29.879Z
paper_authors._id: 646c39fc3e2a7b06594bac6c
paper_authors.avatarUrl: /avatars/8925fef2b6e7105a0f0a4bbf160cfda3.svg
paper_authors.fullname: Hiroki Furuta
paper_authors.type: user
paper_authors.user: frt03
paper_authors._id: 646af37ac37ca1e1230d3505
paper_authors.name: Ofir Nachum
paper_authors._id: 646af37ac37ca1e1230d3506
paper_authors.name: Kuang-Huei Lee
paper_authors._id: 646af37ac37ca1e1230d3507
paper_authors.name: Yutaka Matsuo
paper_authors._id: 646af37ac37ca1e1230d3508
paper_authors.name: Shixiang Shane Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T08:52:31.752Z
paper_authors._id: 631ea400e207d8fe9568dcb2
paper_authors.avatarUrl: /avatars/ab972dc03fcb67ea76378c6b8f7a2622.svg
paper_authors.fullname: Shane Gu
paper_authors.type: user
paper_authors.user: shanegu
paper_authors._id: 646af37ac37ca1e1230d3509
paper_authors.name: Izzeddin Gur
paper_title: Multimodal Web Navigation with Instruction-Finetuned Foundation Models
paper_summary: The progress of autonomous web navigation has been hindered by the dependence
  on billions of exploratory interactions via online reinforcement learning, and
  domain-specific model designs that make it difficult to leverage generalization
  from rich out-of-domain data. In this work, we study data-driven offline
  training for web agents with vision-language foundation models. We propose an
  instruction-following multimodal agent, WebGUM, that observes both webpage
  screenshots and HTML pages and outputs web navigation actions, such as click
  and type. WebGUM is trained by jointly finetuning an instruction-finetuned
  language model and a vision transformer on a large corpus of demonstrations. We
  empirically demonstrate this recipe improves the agent's ability of grounded
  visual perception, HTML comprehension and multi-step reasoning, outperforming
  prior works by a significant margin. On the MiniWoB benchmark, we improve over
  the previous best offline methods by more than 31.9%, being close to reaching
  online-finetuned SoTA. On the WebShop benchmark, our 3-billion-parameter model
  achieves superior performance to the existing SoTA, PaLM-540B. We also collect
  347K high-quality demonstrations using our trained models, 38 times larger than
  prior work, and make them available to promote future research in this
  direction.
paper_ai_keywords: offline training instruction-following multimodal agent WebGUM webpage screenshots HTML pages click type instruction-finetuned language model vision transformer demonstrations grounded visual perception HTML comprehension multi-step reasoning MiniWoB benchmark WebShop benchmark 3-billion-parameter model SoTA (State of the Art) PaLM-540B