title: ZeroGUI: Automating Online GUI Learning at Zero Human Cost
summary: The rapid advancement of large Vision-Language Models (VLMs) has propelled
  the development of pure-vision-based GUI Agents, capable of perceiving and
  operating Graphical User Interfaces (GUI) to autonomously fulfill user
  instructions. However, existing approaches usually adopt an offline learning
  framework, which faces two core limitations: (1) heavy reliance on high-quality
  manual annotations for element grounding and action supervision, and (2)
  limited adaptability to dynamic and interactive environments. To address these
  limitations, we propose ZeroGUI, a scalable, online learning framework for
  automating GUI Agent training at Zero human cost. Specifically, ZeroGUI
  integrates (i) VLM-based automatic task generation to produce diverse training
  goals from the current environment state, (ii) VLM-based automatic reward
  estimation to assess task success without hand-crafted evaluation functions,
  and (iii) two-stage online reinforcement learning to continuously interact with
  and learn from GUI environments. Experiments on two advanced GUI Agents
  (UI-TARS and Aguvis) demonstrate that ZeroGUI significantly boosts performance
  across OSWorld and AndroidLab environments. The code is available at
  https://github.com/OpenGVLab/ZeroGUI.
paper_authors._id: 68391353d8c153d346e1ddb5
paper_authors.name: Chenyu Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:53:52.043Z
paper_authors._id: 637f347a52229c639211bee8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637f347a52229c639211bee8/I9_PET-_6SJQJ6hXrACV4.jpeg
paper_authors.fullname: Chenyu Yang
paper_authors.type: user
paper_authors.user: cyyang822
paper_authors._id: 68391353d8c153d346e1ddb6
paper_authors.name: Shiqian Su
paper_authors._id: 68391353d8c153d346e1ddb7
paper_authors.name: Shi Liu
paper_authors._id: 68391353d8c153d346e1ddb8
paper_authors.name: Xuan Dong
paper_authors._id: 68391353d8c153d346e1ddb9
paper_authors.name: Yue Yu
paper_authors._id: 68391353d8c153d346e1ddba
paper_authors.name: Weijie Su
paper_authors._id: 68391353d8c153d346e1ddbb
paper_authors.name: Xuehui Wang
paper_authors._id: 68391353d8c153d346e1ddbc
paper_authors.name: Zhaoyang Liu
paper_authors._id: 68391353d8c153d346e1ddbd
paper_authors.name: Jinguo Zhu
paper_authors._id: 68391353d8c153d346e1ddbe
paper_authors.name: Hao Li
paper_authors._id: 68391353d8c153d346e1ddbf
paper_authors.name: Wenhai Wang
paper_authors._id: 68391353d8c153d346e1ddc0
paper_authors.name: Yu Qiao
paper_authors._id: 68391353d8c153d346e1ddc1
paper_authors.name: Xizhou Zhu
paper_authors._id: 68391353d8c153d346e1ddc2
paper_authors.name: Jifeng Dai
paper_title: ZeroGUI: Automating Online GUI Learning at Zero Human Cost
paper_summary: The rapid advancement of large Vision-Language Models (VLMs) has propelled
  the development of pure-vision-based GUI Agents, capable of perceiving and
  operating Graphical User Interfaces (GUI) to autonomously fulfill user
  instructions. However, existing approaches usually adopt an offline learning
  framework, which faces two core limitations: (1) heavy reliance on high-quality
  manual annotations for element grounding and action supervision, and (2)
  limited adaptability to dynamic and interactive environments. To address these
  limitations, we propose ZeroGUI, a scalable, online learning framework for
  automating GUI Agent training at Zero human cost. Specifically, ZeroGUI
  integrates (i) VLM-based automatic task generation to produce diverse training
  goals from the current environment state, (ii) VLM-based automatic reward
  estimation to assess task success without hand-crafted evaluation functions,
  and (iii) two-stage online reinforcement learning to continuously interact with
  and learn from GUI environments. Experiments on two advanced GUI Agents
  (UI-TARS and Aguvis) demonstrate that ZeroGUI significantly boosts performance
  across OSWorld and AndroidLab environments. The code is available at
  https://github.com/OpenGVLab/ZeroGUI.
paper_ai_summary: ZeroGUI is an online learning framework that uses Vision-Language Models for task generation and reward estimation, enhancing GUI Agents' performance with minimal human intervention.
paper_ai_keywords: Vision-Language Models GUI Agents element grounding action supervision offline learning automatic task generation automatic reward estimation reinforcement learning OSWorld AndroidLab