WorldGUI: Dynamic Testing for Comprehensive Desktop GUI Automation
Current GUI agents have achieved outstanding performance in GUI element
grounding. However, planning remains highly challenging, especially due to
sensitivity to the initial state of the environment. Specifically, slight
differences in the initial state-such as the target software not being open or
the interface not being in its default state-often lead to planning errors.
This issue is widespread in real user scenarios, but existing benchmarks fail
to evaluate it. In this paper, we present WorldGUI, a novel GUI benchmark that
designs GUI tasks with various initial states to simulate real computer-user
interactions. The benchmark spans a wide range of tasks across 10 popular
software applications, including PowerPoint, VSCode, and Adobe Acrobat. In
addition, to address the challenges of dynamic GUI automation tasks, we propose
GUI-Thinker, a holistic framework, leveraging a critique mechanism, that
effectively manages the unpredictability and complexity of GUI interactions.
Experimental results demonstrate that GUI-Thinker significantly outperforms
Claude-3.5 (Computer Use) by 14.9% in success rate on WorldGUI tasks. This
improvement underscores the effectiveness of our critical-thinking-based
framework in enhancing GUI automation.
67ad92bfbbf3810ab20595c2
Henry Hengyuan Zhao
admin_assigned
2025-02-14T15:30:16.490Z
647d7eb9770c299e56f5b39b
https://cdn-avatars.huggingface.co/v1/production/uploads/647d7eb9770c299e56f5b39b/CC5JJgkyLkXOxw-BeT4G5.jpeg
Henry Hengyuan Zhao
user
hhenryz
67ad92bfbbf3810ab20595c3
Difei Gao
admin_assigned
2025-02-14T15:30:23.319Z
661abfba70ec567e154fbb00
https://cdn-avatars.huggingface.co/v1/production/uploads/661abfba70ec567e154fbb00/-0EDBVhWSL8_O1ynSwIzG.jpeg
Difei Gao
user
QuStar
67ad92bfbbf3810ab20595c4
Mike Zheng Shou
admin_assigned
2025-02-14T15:30:30.271Z
661ab3da2b14565c7acccf5c
/avatars/fa4fc03664803e02aede4d4c3d50b393.svg
Mike Zheng Shou
user
AnalMom
WorldGUI: Dynamic Testing for Comprehensive Desktop GUI Automation
Current GUI agents have achieved outstanding performance in GUI element
grounding. However, planning remains highly challenging, especially due to
sensitivity to the initial state of the environment. Specifically, slight
differences in the initial state-such as the target software not being open or
the interface not being in its default state-often lead to planning errors.
This issue is widespread in real user scenarios, but existing benchmarks fail
to evaluate it. In this paper, we present WorldGUI, a novel GUI benchmark that
designs GUI tasks with various initial states to simulate real computer-user
interactions. The benchmark spans a wide range of tasks across 10 popular
software applications, including PowerPoint, VSCode, and Adobe Acrobat. In
addition, to address the challenges of dynamic GUI automation tasks, we propose
GUI-Thinker, a holistic framework, leveraging a critique mechanism, that
effectively manages the unpredictability and complexity of GUI interactions.
Experimental results demonstrate that GUI-Thinker significantly outperforms
Claude-3.5 (Computer Use) by 14.9% in success rate on WorldGUI tasks. This
improvement underscores the effectiveness of our critical-thinking-based
framework in enhancing GUI automation.
The paper presents WorldGUI, a novel benchmark for GUI tasks with varying initial states, and GUI-Thinker, a critique mechanism-based framework, which together improve GUI automation performance significantly.
GUI element grounding GUI benchmark dynamic GUI automation critique mechanism GUI-Thinker