title: WorldGUI: Dynamic Testing for Comprehensive Desktop GUI Automation
summary: Current GUI agents have achieved outstanding performance in GUI element
  grounding. However, planning remains highly challenging, especially due to
  sensitivity to the initial state of the environment. Specifically, slight
  differences in the initial state-such as the target software not being open or
  the interface not being in its default state-often lead to planning errors.
  This issue is widespread in real user scenarios, but existing benchmarks fail
  to evaluate it. In this paper, we present WorldGUI, a novel GUI benchmark that
  designs GUI tasks with various initial states to simulate real computer-user
  interactions. The benchmark spans a wide range of tasks across 10 popular
  software applications, including PowerPoint, VSCode, and Adobe Acrobat. In
  addition, to address the challenges of dynamic GUI automation tasks, we propose
  GUI-Thinker, a holistic framework, leveraging a critique mechanism, that
  effectively manages the unpredictability and complexity of GUI interactions.
  Experimental results demonstrate that GUI-Thinker significantly outperforms
  Claude-3.5 (Computer Use) by 14.9% in success rate on WorldGUI tasks. This
  improvement underscores the effectiveness of our critical-thinking-based
  framework in enhancing GUI automation.
paper_authors._id: 67ad92bfbbf3810ab20595c2
paper_authors.name: Henry Hengyuan Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T15:30:16.490Z
paper_authors._id: 647d7eb9770c299e56f5b39b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647d7eb9770c299e56f5b39b/CC5JJgkyLkXOxw-BeT4G5.jpeg
paper_authors.fullname: Henry Hengyuan Zhao
paper_authors.type: user
paper_authors.user: hhenryz
paper_authors._id: 67ad92bfbbf3810ab20595c3
paper_authors.name: Difei Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T15:30:23.319Z
paper_authors._id: 661abfba70ec567e154fbb00
paper_authors.avatarUrl: /avatars/46c004650ba15525ccf02d40c9314085.svg
paper_authors.fullname: Difei Gao
paper_authors.type: user
paper_authors.user: QuStar
paper_authors._id: 67ad92bfbbf3810ab20595c4
paper_authors.name: Mike Zheng Shou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-14T15:30:30.271Z
paper_authors._id: 661ab3da2b14565c7acccf5c
paper_authors.avatarUrl: /avatars/fa4fc03664803e02aede4d4c3d50b393.svg
paper_authors.fullname: Mike Zheng Shou
paper_authors.type: user
paper_authors.user: AnalMom
paper_title: WorldGUI: Dynamic Testing for Comprehensive Desktop GUI Automation
paper_summary: Current GUI agents have achieved outstanding performance in GUI element
  grounding. However, planning remains highly challenging, especially due to
  sensitivity to the initial state of the environment. Specifically, slight
  differences in the initial state-such as the target software not being open or
  the interface not being in its default state-often lead to planning errors.
  This issue is widespread in real user scenarios, but existing benchmarks fail
  to evaluate it. In this paper, we present WorldGUI, a novel GUI benchmark that
  designs GUI tasks with various initial states to simulate real computer-user
  interactions. The benchmark spans a wide range of tasks across 10 popular
  software applications, including PowerPoint, VSCode, and Adobe Acrobat. In
  addition, to address the challenges of dynamic GUI automation tasks, we propose
  GUI-Thinker, a holistic framework, leveraging a critique mechanism, that
  effectively manages the unpredictability and complexity of GUI interactions.
  Experimental results demonstrate that GUI-Thinker significantly outperforms
  Claude-3.5 (Computer Use) by 14.9% in success rate on WorldGUI tasks. This
  improvement underscores the effectiveness of our critical-thinking-based
  framework in enhancing GUI automation.
paper_ai_keywords: GUI agents GUI element grounding planning initial state WorldGUI GUI benchmark GUI tasks software applications PowerPoint VSCode Adobe Acrobat GUI automation tasks GUI-Thinker critique mechanism success rate