title: Navigating the Digital World as Humans Do: Universal Visual Grounding
    for GUI Agents
summary: Multimodal large language models (MLLMs) are transforming the capabilities of
  graphical user interface (GUI) agents, facilitating their transition from
  controlled simulations to complex, real-world applications across various
  platforms. However, the effectiveness of these agents hinges on the robustness
  of their grounding capability. Current GUI agents predominantly utilize
  text-based representations such as HTML or accessibility trees, which, despite
  their utility, often introduce noise, incompleteness, and increased
  computational overhead. In this paper, we advocate a human-like embodiment for
  GUI agents that perceive the environment entirely visually and directly take
  pixel-level operations on the GUI. The key is visual grounding models that can
  accurately map diverse referring expressions of GUI elements to their
  coordinates on the GUI across different platforms. We show that a simple
  recipe, which includes web-based synthetic data and slight adaptation of the
  LLaVA architecture, is surprisingly effective for training such visual
  grounding models. We collect the largest dataset for GUI visual grounding so
  far, containing 10M GUI elements and their referring expressions over 1.3M
  screenshots, and use it to train UGround, a strong universal visual grounding
  model for GUI agents. Empirical results on six benchmarks spanning three
  categories (grounding, offline agent, and online agent) show that 1) UGround
  substantially outperforms existing visual grounding models for GUI agents, by
  up to 20% absolute, and 2) agents with UGround outperform state-of-the-art
  agents, despite the fact that existing agents use additional text-based input
  while ours only uses visual perception. These results provide strong support
  for the feasibility and promises of GUI agents that navigate the digital world
  as humans do.
paper_authors._id: 6704ac58fdaeb32c253487c2
paper_authors.name: Boyu Gou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T15:17:32.360Z
paper_authors._id: 6500870f1e14749e84f8f887
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6500870f1e14749e84f8f887/wfvx4BZvh2OyW-vpq5jEy.jpeg
paper_authors.fullname: Boyu Gou
paper_authors.type: user
paper_authors.user: BoyuNLP
paper_authors._id: 6704ac58fdaeb32c253487c3
paper_authors.name: Ruohan Wang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-08T03:51:54.906Z
paper_authors._id: 649d00587016f9f2fcbc0fd9
paper_authors.avatarUrl: /avatars/4f2e3e239bb19c1597463f2d5010fb7d.svg
paper_authors.fullname: Yanan Xie
paper_authors.type: user
paper_authors.user: orby-yanan
paper_authors._id: 6704ac58fdaeb32c253487c4
paper_authors.name: Boyuan Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:02:04.746Z
paper_authors._id: 631a95cfa66151e36e54a905
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631a95cfa66151e36e54a905/jM_4vy9_Kd8zomFVYtsTo.png
paper_authors.fullname: Boyuan Zheng
paper_authors.type: user
paper_authors.user: boyuanzheng010
paper_authors._id: 6704ac58fdaeb32c253487c5
paper_authors.name: Yanan Xie
paper_authors._id: 6704ac58fdaeb32c253487c6
paper_authors.name: Cheng Chang
paper_authors._id: 6704ac58fdaeb32c253487c7
paper_authors.name: Yiheng Shu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T15:17:28.570Z
paper_authors._id: 60a4ebfbaa9320dbbe69e37c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60a4ebfbaa9320dbbe69e37c/QLaEohXCWaUy8YX3wKQ_w.jpeg
paper_authors.fullname: Yiheng Shu
paper_authors.type: user
paper_authors.user: yhshu
paper_authors._id: 6704ac58fdaeb32c253487c8
paper_authors.name: Huan Sun
paper_authors._id: 6704ac58fdaeb32c253487c9
paper_authors.name: Yu Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T15:17:30.464Z
paper_authors._id: 6477a323dbc2a416f8b852b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
paper_authors.fullname: Yu Su
paper_authors.type: user
paper_authors.user: ysu-nlp
paper_title: Navigating the Digital World as Humans Do: Universal Visual Grounding
    for GUI Agents
paper_summary: Multimodal large language models (MLLMs) are transforming the capabilities of
  graphical user interface (GUI) agents, facilitating their transition from
  controlled simulations to complex, real-world applications across various
  platforms. However, the effectiveness of these agents hinges on the robustness
  of their grounding capability. Current GUI agents predominantly utilize
  text-based representations such as HTML or accessibility trees, which, despite
  their utility, often introduce noise, incompleteness, and increased
  computational overhead. In this paper, we advocate a human-like embodiment for
  GUI agents that perceive the environment entirely visually and directly take
  pixel-level operations on the GUI. The key is visual grounding models that can
  accurately map diverse referring expressions of GUI elements to their
  coordinates on the GUI across different platforms. We show that a simple
  recipe, which includes web-based synthetic data and slight adaptation of the
  LLaVA architecture, is surprisingly effective for training such visual
  grounding models. We collect the largest dataset for GUI visual grounding so
  far, containing 10M GUI elements and their referring expressions over 1.3M
  screenshots, and use it to train UGround, a strong universal visual grounding
  model for GUI agents. Empirical results on six benchmarks spanning three
  categories (grounding, offline agent, and online agent) show that 1) UGround
  substantially outperforms existing visual grounding models for GUI agents, by
  up to 20% absolute, and 2) agents with UGround outperform state-of-the-art
  agents, despite the fact that existing agents use additional text-based input
  while ours only uses visual perception. These results provide strong support
  for the feasibility and promises of GUI agents that navigate the digital world
  as humans do.
paper_ai_summary: Visual grounding models for GUI agents using pixel-level operations outperform text-based methods, demonstrating the feasibility of human-like visual perception in complex applications.
paper_ai_keywords: multimodal large language models GUI agents visual grounding models pixel-level operations referring expressions synthetic data LLaVA architecture universal visual grounding model GUI visual grounding dataset offline agent online agent