GTA1: GUI Test-time Scaling Agent
Graphical user interface (GUI) agents autonomously operate across platforms
(e.g., Linux) to complete tasks by interacting with visual elements.
Specifically, a user instruction is decomposed into a sequence of action
proposals, each corresponding to an interaction with the GUI. After each
action, the agent observes the updated GUI environment to plan the next step.
However, two main challenges arise: i) resolving ambiguity in task planning
(i.e., the action proposal sequence), where selecting an appropriate plan is
non-trivial, as many valid ones may exist; ii) accurately grounding actions in
complex and high-resolution interfaces, i.e., precisely interacting with visual
targets.
  This paper investigates the two aforementioned challenges with our GUI
Test-time Scaling Agent, namely GTA1. First, to select the most appropriate
action proposal, we introduce a test-time scaling method. At each step, we
sample multiple candidate action proposals and leverage a judge model to
evaluate and select the most suitable one. It trades off computation for better
decision quality by concurrent sampling, shortening task execution steps, and
improving overall performance. Second, we propose a model that achieves
improved accuracy when grounding the selected action proposal to its
corresponding visual elements. Our key insight is that reinforcement learning
(RL) facilitates visual grounding through inherent objective alignments,
rewarding successful clicks on interface elements.
  Experimentally, our method establishes state-of-the-art performance across
diverse benchmarks. For example, GTA1-7B achieves 50.1%, 92.4%, and 67.7%
accuracies on Screenspot-Pro, Screenspot-V2, and OSWorld-G, respectively. When
paired with a planner applying our test-time scaling strategy, it exhibits
state-of-the-art agentic performance (e.g., 45.2% task success rate on
OSWorld). We open-source our code and models here.
686dcbe6cb5725779c60b37f
Yan Yang
686dcbe6cb5725779c60b380
Dongxu Li
686dcbe6cb5725779c60b381
Yutong Dai
686dcbe6cb5725779c60b382
Yuhao Yang
686dcbe6cb5725779c60b383
Ziyang Luo
claimed_verified
2025-07-09T08:50:27.741Z
6090ff099a8bcaa437b234a4
https://cdn-avatars.huggingface.co/v1/production/uploads/6090ff099a8bcaa437b234a4/iUvw7JXT-ngI7rGk1x-io.jpeg
Ziyang Luo
user
Ziyang
686dcbe6cb5725779c60b384
Zirui Zhao
686dcbe6cb5725779c60b385
Zhiyuan Hu
686dcbe6cb5725779c60b386
Junzhe Huang
686dcbe6cb5725779c60b387
Amrita Saha
686dcbe6cb5725779c60b388
Zeyuan Chen
686dcbe6cb5725779c60b389
Ran Xu
686dcbe6cb5725779c60b38a
Liyuan Pan
686dcbe6cb5725779c60b38b
Caiming Xiong
686dcbe6cb5725779c60b38c
Junnan Li
GTA1: GUI Test-time Scaling Agent
Graphical user interface (GUI) agents autonomously operate across platforms
(e.g., Linux) to complete tasks by interacting with visual elements.
Specifically, a user instruction is decomposed into a sequence of action
proposals, each corresponding to an interaction with the GUI. After each
action, the agent observes the updated GUI environment to plan the next step.
However, two main challenges arise: i) resolving ambiguity in task planning
(i.e., the action proposal sequence), where selecting an appropriate plan is
non-trivial, as many valid ones may exist; ii) accurately grounding actions in
complex and high-resolution interfaces, i.e., precisely interacting with visual
targets.
  This paper investigates the two aforementioned challenges with our GUI
Test-time Scaling Agent, namely GTA1. First, to select the most appropriate
action proposal, we introduce a test-time scaling method. At each step, we
sample multiple candidate action proposals and leverage a judge model to
evaluate and select the most suitable one. It trades off computation for better
decision quality by concurrent sampling, shortening task execution steps, and
improving overall performance. Second, we propose a model that achieves
improved accuracy when grounding the selected action proposal to its
corresponding visual elements. Our key insight is that reinforcement learning
(RL) facilitates visual grounding through inherent objective alignments,
rewarding successful clicks on interface elements.
  Experimentally, our method establishes state-of-the-art performance across
diverse benchmarks. For example, GTA1-7B achieves 50.1%, 92.4%, and 67.7%
accuracies on Screenspot-Pro, Screenspot-V2, and OSWorld-G, respectively. When
paired with a planner applying our test-time scaling strategy, it exhibits
state-of-the-art agentic performance (e.g., 45.2% task success rate on
OSWorld). We open-source our code and models here.
A GUI Test-time Scaling Agent addresses task planning ambiguity and visual grounding accuracy in GUI interactions using reinforcement learning and test-time scaling.
GUI action proposals test-time scaling judge model reinforcement learning visual grounding Screenspot-Pro Screenspot-V2 OSWorld-G