title: GPT-4V(ision) is a Generalist Web Agent, if Grounded
summary: The recent development on large multimodal models (LMMs), especially
  GPT-4V(ision) and Gemini, has been quickly expanding the capability boundaries
  of multimodal models beyond traditional tasks like image captioning and visual
  question answering. In this work, we explore the potential of LMMs like GPT-4V
  as a generalist web agent that can follow natural language instructions to
  complete tasks on any given website. We propose SEEACT, a generalist web agent
  that harnesses the power of LMMs for integrated visual understanding and acting
  on the web. We evaluate on the recent MIND2WEB benchmark. In addition to
  standard offline evaluation on cached websites, we enable a new online
  evaluation setting by developing a tool that allows running web agents on live
  websites. We show that GPT-4V presents a great potential for web agents - it
  can successfully complete 50% of the tasks on live websites if we manually
  ground its textual plans into actions on the websites. This substantially
  outperforms text-only LLMs like GPT-4 or smaller models (FLAN-T5 and BLIP-2)
  specifically fine-tuned for web agents. However, grounding still remains a
  major challenge. Existing LMM grounding strategies like set-of-mark prompting
  turns out not effective for web agents, and the best grounding strategy we
  develop in this paper leverages both the HTML text and visuals. Yet, there is
  still a substantial gap with oracle grounding, leaving ample room for further
  improvement.
paper_authors._id: 65960e11b02e572eb0bfd00c
paper_authors.name: Boyuan Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-18T08:37:29.919Z
paper_authors._id: 631a95cfa66151e36e54a905
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631a95cfa66151e36e54a905/jM_4vy9_Kd8zomFVYtsTo.png
paper_authors.fullname: Boyuan Zheng
paper_authors.type: user
paper_authors.user: boyuanzheng010
paper_authors._id: 65960e11b02e572eb0bfd00d
paper_authors.name: Boyu Gou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-04T09:35:11.800Z
paper_authors._id: 6500870f1e14749e84f8f887
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6500870f1e14749e84f8f887/wfvx4BZvh2OyW-vpq5jEy.jpeg
paper_authors.fullname: Boyu Gou
paper_authors.type: user
paper_authors.user: BoyuNLP
paper_authors._id: 65960e11b02e572eb0bfd00e
paper_authors.name: Jihyung Kil
paper_authors._id: 65960e11b02e572eb0bfd00f
paper_authors.name: Huan Sun
paper_authors._id: 65960e11b02e572eb0bfd010
paper_authors.name: Yu Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-04T09:35:14.490Z
paper_authors._id: 6477a323dbc2a416f8b852b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/grnmnxO1foz6aRRJ_A2NU.jpeg
paper_authors.fullname: Yu Su
paper_authors.type: user
paper_authors.user: ysu-nlp
paper_title: GPT-4V(ision) is a Generalist Web Agent, if Grounded
paper_summary: The recent development on large multimodal models (LMMs), especially
  GPT-4V(ision) and Gemini, has been quickly expanding the capability boundaries
  of multimodal models beyond traditional tasks like image captioning and visual
  question answering. In this work, we explore the potential of LMMs like GPT-4V
  as a generalist web agent that can follow natural language instructions to
  complete tasks on any given website. We propose SEEACT, a generalist web agent
  that harnesses the power of LMMs for integrated visual understanding and acting
  on the web. We evaluate on the recent MIND2WEB benchmark. In addition to
  standard offline evaluation on cached websites, we enable a new online
  evaluation setting by developing a tool that allows running web agents on live
  websites. We show that GPT-4V presents a great potential for web agents - it
  can successfully complete 50% of the tasks on live websites if we manually
  ground its textual plans into actions on the websites. This substantially
  outperforms text-only LLMs like GPT-4 or smaller models (FLAN-T5 and BLIP-2)
  specifically fine-tuned for web agents. However, grounding still remains a
  major challenge. Existing LMM grounding strategies like set-of-mark prompting
  turns out not effective for web agents, and the best grounding strategy we
  develop in this paper leverages both the HTML text and visuals. Yet, there is
  still a substantial gap with oracle grounding, leaving ample room for further
  improvement.
paper_ai_keywords: large multimodal models (LMMs) GPT-4V(ision) Gemini multimodal models image captioning visual question answering SEEACT integrated visual understanding MIND2WEB benchmark online evaluation web agents GPT-4 text-only LLMs FLAN-T5 BLIP-2 set-of-mark prompting oracle grounding