ZeroGUI: Automating Online GUI Learning at Zero Human Cost
The rapid advancement of large Vision-Language Models (VLMs) has propelled
the development of pure-vision-based GUI Agents, capable of perceiving and
operating Graphical User Interfaces (GUI) to autonomously fulfill user
instructions. However, existing approaches usually adopt an offline learning
framework, which faces two core limitations: (1) heavy reliance on high-quality
manual annotations for element grounding and action supervision, and (2)
limited adaptability to dynamic and interactive environments. To address these
limitations, we propose ZeroGUI, a scalable, online learning framework for
automating GUI Agent training at Zero human cost. Specifically, ZeroGUI
integrates (i) VLM-based automatic task generation to produce diverse training
goals from the current environment state, (ii) VLM-based automatic reward
estimation to assess task success without hand-crafted evaluation functions,
and (iii) two-stage online reinforcement learning to continuously interact with
and learn from GUI environments. Experiments on two advanced GUI Agents
(UI-TARS and Aguvis) demonstrate that ZeroGUI significantly boosts performance
across OSWorld and AndroidLab environments. The code is available at
https://github.com/OpenGVLab/ZeroGUI.
68391353d8c153d346e1ddb5
Chenyu Yang
claimed_verified
2025-05-30T06:53:52.043Z
637f347a52229c639211bee8
https://cdn-avatars.huggingface.co/v1/production/uploads/637f347a52229c639211bee8/I9_PET-_6SJQJ6hXrACV4.jpeg
Chenyu Yang
user
cyyang822
68391353d8c153d346e1ddb6
Shiqian Su
68391353d8c153d346e1ddb7
Shi Liu
claimed_verified
2025-06-18T12:17:36.942Z
637f22d27119bd030dfd4af8
/avatars/55c468c40ad3bd3218d086759fb1be3c.svg
Shi Liu
user
CLLBJ16
68391353d8c153d346e1ddb8
Xuan Dong
68391353d8c153d346e1ddb9
Yue Yu
68391353d8c153d346e1ddba
Weijie Su
claimed_verified
2025-06-19T09:10:48.831Z
63e4562f9db5da2dc1f3b520
/avatars/f4eecf1396b05e1c72436e7026d85cef.svg
Weijie Su
user
jackroos
68391353d8c153d346e1ddbb
Xuehui Wang
68391353d8c153d346e1ddbc
Zhaoyang Liu
68391353d8c153d346e1ddbd
Jinguo Zhu
68391353d8c153d346e1ddbe
Hao Li
68391353d8c153d346e1ddbf
Wenhai Wang
68391353d8c153d346e1ddc0
Yu Qiao
68391353d8c153d346e1ddc1
Xizhou Zhu
68391353d8c153d346e1ddc2
Jifeng Dai
ZeroGUI: Automating Online GUI Learning at Zero Human Cost
The rapid advancement of large Vision-Language Models (VLMs) has propelled
the development of pure-vision-based GUI Agents, capable of perceiving and
operating Graphical User Interfaces (GUI) to autonomously fulfill user
instructions. However, existing approaches usually adopt an offline learning
framework, which faces two core limitations: (1) heavy reliance on high-quality
manual annotations for element grounding and action supervision, and (2)
limited adaptability to dynamic and interactive environments. To address these
limitations, we propose ZeroGUI, a scalable, online learning framework for
automating GUI Agent training at Zero human cost. Specifically, ZeroGUI
integrates (i) VLM-based automatic task generation to produce diverse training
goals from the current environment state, (ii) VLM-based automatic reward
estimation to assess task success without hand-crafted evaluation functions,
and (iii) two-stage online reinforcement learning to continuously interact with
and learn from GUI environments. Experiments on two advanced GUI Agents
(UI-TARS and Aguvis) demonstrate that ZeroGUI significantly boosts performance
across OSWorld and AndroidLab environments. The code is available at
https://github.com/OpenGVLab/ZeroGUI.
ZeroGUI is an online learning framework that uses Vision-Language Models for task generation and reward estimation, enhancing GUI Agents' performance with minimal human intervention.
Vision-Language Models GUI Agents element grounding action supervision offline learning automatic task generation automatic reward estimation reinforcement learning OSWorld AndroidLab