title: GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
summary: One of the principal challenges in building VLM-powered GUI agents is visual
  grounding, i.e., localizing the appropriate screen region for action execution
  based on both the visual content and the textual plans. Most existing work
  formulates this as a text-based coordinate generation task. However, these
  approaches suffer from several limitations: weak spatial-semantic alignment,
  inability to handle ambiguous supervision targets, and a mismatch between the
  dense nature of screen coordinates and the coarse, patch-level granularity of
  visual features extracted by models like Vision Transformers. In this paper, we
  propose GUI-Actor, a VLM-based method for coordinate-free GUI grounding. At its
  core, GUI-Actor introduces an attention-based action head that learns to align
  a dedicated <ACTOR> token with all relevant visual patch tokens, enabling the
  model to propose one or more action regions in a single forward pass. In line
  with this, we further design a grounding verifier to evaluate and select the
  most plausible action region from the candidates proposed for action execution.
  Extensive experiments show that GUI-Actor outperforms prior state-of-the-art
  methods on multiple GUI action grounding benchmarks, with improved
  generalization to unseen screen resolutions and layouts. Notably, GUI-Actor-7B
  even surpasses UI-TARS-72B (38.1) on ScreenSpot-Pro, achieving scores of 40.7
  with Qwen2-VL and 44.6 with Qwen2.5-VL as backbones. Furthermore, by
  incorporating the verifier, we find that fine-tuning only the newly introduced
  action head (~100M parameters for 7B model) while keeping the VLM backbone
  frozen is sufficient to achieve performance comparable to previous
  state-of-the-art models, highlighting that GUI-Actor can endow the underlying
  VLM with effective grounding capabilities without compromising its
  general-purpose strengths.
paper_authors._id: 683fc5599363b50c19f17d42
paper_authors.name: Qianhui Wu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-06-04T04:21:31.171Z
paper_authors._id: 63ef330b1e695b35aa484e11
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ef330b1e695b35aa484e11/bXwpGy0dl8JXeJwJ--ilr.jpeg
paper_authors.fullname: Qianhui WU
paper_authors.type: user
paper_authors.user: qianhuiwu
paper_authors._id: 683fc5599363b50c19f17d43
paper_authors.name: Kanzhi Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T08:54:48.476Z
paper_authors._id: 63340dbbd92c5842ae71d1e9
paper_authors.avatarUrl: /avatars/3a3182996bd41b526dcbfa8687d91963.svg
paper_authors.fullname: Kanzhi Cheng
paper_authors.type: user
paper_authors.user: cckevinn
paper_authors._id: 683fc5599363b50c19f17d44
paper_authors.name: Rui Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T08:54:51.070Z
paper_authors._id: 64d45451c34a346181b130dd
paper_authors.avatarUrl: /avatars/9bb8205b889337df5d321539c9b5d69d.svg
paper_authors.fullname: Rui Yang
paper_authors.type: user
paper_authors.user: Ray2333
paper_authors._id: 683fc5599363b50c19f17d45
paper_authors.name: Chaoyun Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T08:54:54.267Z
paper_authors._id: 654dbac9938fbf1e696be8aa
paper_authors.avatarUrl: /avatars/b3c4035c48169c1bfb04a439fce3499f.svg
paper_authors.fullname: Chaoyun Zhang
paper_authors.type: user
paper_authors.user: vyokky
paper_authors._id: 683fc5599363b50c19f17d46
paper_authors.name: Jianwei Yang
paper_authors._id: 683fc5599363b50c19f17d47
paper_authors.name: Huiqiang Jiang
paper_authors._id: 683fc5599363b50c19f17d48
paper_authors.name: Jian Mu
paper_authors._id: 683fc5599363b50c19f17d49
paper_authors.name: Baolin Peng
paper_authors._id: 683fc5599363b50c19f17d4a
paper_authors.name: Bo Qiao
paper_authors._id: 683fc5599363b50c19f17d4b
paper_authors.name: Reuben Tan
paper_authors._id: 683fc5599363b50c19f17d4c
paper_authors.name: Si Qin
paper_authors._id: 683fc5599363b50c19f17d4d
paper_authors.name: Lars Liden
paper_authors._id: 683fc5599363b50c19f17d4e
paper_authors.name: Qingwei Lin
paper_authors._id: 683fc5599363b50c19f17d4f
paper_authors.name: Huan Zhang
paper_authors._id: 683fc5599363b50c19f17d50
paper_authors.name: Tong Zhang
paper_authors._id: 683fc5599363b50c19f17d51
paper_authors.name: Jianbing Zhang
paper_authors._id: 683fc5599363b50c19f17d52
paper_authors.name: Dongmei Zhang
paper_authors._id: 683fc5599363b50c19f17d53
paper_authors.name: Jianfeng Gao
paper_title: GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
paper_summary: One of the principal challenges in building VLM-powered GUI agents is visual
  grounding, i.e., localizing the appropriate screen region for action execution
  based on both the visual content and the textual plans. Most existing work
  formulates this as a text-based coordinate generation task. However, these
  approaches suffer from several limitations: weak spatial-semantic alignment,
  inability to handle ambiguous supervision targets, and a mismatch between the
  dense nature of screen coordinates and the coarse, patch-level granularity of
  visual features extracted by models like Vision Transformers. In this paper, we
  propose GUI-Actor, a VLM-based method for coordinate-free GUI grounding. At its
  core, GUI-Actor introduces an attention-based action head that learns to align
  a dedicated <ACTOR> token with all relevant visual patch tokens, enabling the
  model to propose one or more action regions in a single forward pass. In line
  with this, we further design a grounding verifier to evaluate and select the
  most plausible action region from the candidates proposed for action execution.
  Extensive experiments show that GUI-Actor outperforms prior state-of-the-art
  methods on multiple GUI action grounding benchmarks, with improved
  generalization to unseen screen resolutions and layouts. Notably, GUI-Actor-7B
  even surpasses UI-TARS-72B (38.1) on ScreenSpot-Pro, achieving scores of 40.7
  with Qwen2-VL and 44.6 with Qwen2.5-VL as backbones. Furthermore, by
  incorporating the verifier, we find that fine-tuning only the newly introduced
  action head (~100M parameters for 7B model) while keeping the VLM backbone
  frozen is sufficient to achieve performance comparable to previous
  state-of-the-art models, highlighting that GUI-Actor can endow the underlying
  VLM with effective grounding capabilities without compromising its
  general-purpose strengths.
paper_ai_summary: GUI-Actor, a VLM-based method using attention for coordinate-free GUI grounding, outperforms existing methods with better generalization and efficient fine-tuning.
paper_ai_keywords: visual grounding text-based coordinate generation spatial-semantic alignment Vision Transformers attention-based action head grounding verifier GUI action grounding benchmarks GUI-Actor GUI-Actor-7B UI-TARS-72B ScreenSpot-Pro Qwen2-VL Qwen2.5-VL parameter-efficient fine-tuning