GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
One of the principal challenges in building VLM-powered GUI agents is visual
grounding, i.e., localizing the appropriate screen region for action execution
based on both the visual content and the textual plans. Most existing work
formulates this as a text-based coordinate generation task. However, these
approaches suffer from several limitations: weak spatial-semantic alignment,
inability to handle ambiguous supervision targets, and a mismatch between the
dense nature of screen coordinates and the coarse, patch-level granularity of
visual features extracted by models like Vision Transformers. In this paper, we
propose GUI-Actor, a VLM-based method for coordinate-free GUI grounding. At its
core, GUI-Actor introduces an attention-based action head that learns to align
a dedicated <ACTOR> token with all relevant visual patch tokens, enabling the
model to propose one or more action regions in a single forward pass. In line
with this, we further design a grounding verifier to evaluate and select the
most plausible action region from the candidates proposed for action execution.
Extensive experiments show that GUI-Actor outperforms prior state-of-the-art
methods on multiple GUI action grounding benchmarks, with improved
generalization to unseen screen resolutions and layouts. Notably, GUI-Actor-7B
even surpasses UI-TARS-72B (38.1) on ScreenSpot-Pro, achieving scores of 40.7
with Qwen2-VL and 44.6 with Qwen2.5-VL as backbones. Furthermore, by
incorporating the verifier, we find that fine-tuning only the newly introduced
action head (~100M parameters for 7B model) while keeping the VLM backbone
frozen is sufficient to achieve performance comparable to previous
state-of-the-art models, highlighting that GUI-Actor can endow the underlying
VLM with effective grounding capabilities without compromising its
general-purpose strengths.
683fc5599363b50c19f17d42
Qianhui Wu
extracted_confirmed
2025-06-04T04:21:31.171Z
63ef330b1e695b35aa484e11
https://cdn-avatars.huggingface.co/v1/production/uploads/63ef330b1e695b35aa484e11/bXwpGy0dl8JXeJwJ--ilr.jpeg
Qianhui WU
user
qianhuiwu
683fc5599363b50c19f17d43
Kanzhi Cheng
claimed_verified
2025-06-04T08:54:48.476Z
63340dbbd92c5842ae71d1e9
/avatars/3a3182996bd41b526dcbfa8687d91963.svg
Kanzhi Cheng
user
cckevinn
683fc5599363b50c19f17d44
Rui Yang
claimed_verified
2025-06-04T08:54:51.070Z
64d45451c34a346181b130dd
/avatars/9bb8205b889337df5d321539c9b5d69d.svg
Rui Yang
user
Ray2333
683fc5599363b50c19f17d45
Chaoyun Zhang
claimed_verified
2025-06-04T08:54:54.267Z
654dbac9938fbf1e696be8aa
/avatars/b3c4035c48169c1bfb04a439fce3499f.svg
Chaoyun Zhang
user
vyokky
683fc5599363b50c19f17d46
Jianwei Yang
683fc5599363b50c19f17d47
Huiqiang Jiang
683fc5599363b50c19f17d48
Jian Mu
683fc5599363b50c19f17d49
Baolin Peng
683fc5599363b50c19f17d4a
Bo Qiao
683fc5599363b50c19f17d4b
Reuben Tan
683fc5599363b50c19f17d4c
Si Qin
683fc5599363b50c19f17d4d
Lars Liden
683fc5599363b50c19f17d4e
Qingwei Lin
683fc5599363b50c19f17d4f
Huan Zhang
683fc5599363b50c19f17d50
Tong Zhang
683fc5599363b50c19f17d51
Jianbing Zhang
683fc5599363b50c19f17d52
Dongmei Zhang
683fc5599363b50c19f17d53
Jianfeng Gao
GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
One of the principal challenges in building VLM-powered GUI agents is visual
grounding, i.e., localizing the appropriate screen region for action execution
based on both the visual content and the textual plans. Most existing work
formulates this as a text-based coordinate generation task. However, these
approaches suffer from several limitations: weak spatial-semantic alignment,
inability to handle ambiguous supervision targets, and a mismatch between the
dense nature of screen coordinates and the coarse, patch-level granularity of
visual features extracted by models like Vision Transformers. In this paper, we
propose GUI-Actor, a VLM-based method for coordinate-free GUI grounding. At its
core, GUI-Actor introduces an attention-based action head that learns to align
a dedicated <ACTOR> token with all relevant visual patch tokens, enabling the
model to propose one or more action regions in a single forward pass. In line
with this, we further design a grounding verifier to evaluate and select the
most plausible action region from the candidates proposed for action execution.
Extensive experiments show that GUI-Actor outperforms prior state-of-the-art
methods on multiple GUI action grounding benchmarks, with improved
generalization to unseen screen resolutions and layouts. Notably, GUI-Actor-7B
even surpasses UI-TARS-72B (38.1) on ScreenSpot-Pro, achieving scores of 40.7
with Qwen2-VL and 44.6 with Qwen2.5-VL as backbones. Furthermore, by
incorporating the verifier, we find that fine-tuning only the newly introduced
action head (~100M parameters for 7B model) while keeping the VLM backbone
frozen is sufficient to achieve performance comparable to previous
state-of-the-art models, highlighting that GUI-Actor can endow the underlying
VLM with effective grounding capabilities without compromising its
general-purpose strengths.
GUI-Actor, a VLM-based method using attention for coordinate-free GUI grounding, outperforms existing methods with better generalization and efficient fine-tuning.
visual grounding text-based coordinate generation spatial-semantic alignment Vision Transformers attention-based action head grounding verifier GUI action grounding benchmarks GUI-Actor GUI-Actor-7B UI-TARS-72B ScreenSpot-Pro Qwen2-VL Qwen2.5-VL parameter-efficient fine-tuning