OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse
  Task Synthesis
Graphical User Interface (GUI) agents powered by Vision-Language Models
(VLMs) have demonstrated human-like computer control capability. Despite their
utility in advancing digital automation, a critical bottleneck persists:
collecting high-quality trajectory data for training. Common practices for
collecting such data rely on human supervision or synthetic data generation
through executing pre-defined tasks, which are either resource-intensive or
unable to guarantee data quality. Moreover, these methods suffer from limited
data diversity and significant gaps between synthetic data and real-world
environments. To address these challenges, we propose OS-Genesis, a novel GUI
data synthesis pipeline that reverses the conventional trajectory collection
process. Instead of relying on pre-defined tasks, OS-Genesis enables agents
first to perceive environments and perform step-wise interactions, then
retrospectively derive high-quality tasks to enable trajectory-level
exploration. A trajectory reward model is then employed to ensure the quality
of the generated trajectories. We demonstrate that training GUI agents with
OS-Genesis significantly improves their performance on highly challenging
online benchmarks. In-depth analysis further validates OS-Genesis's efficiency
and its superior data quality and diversity compared to existing synthesis
methods. Our codes, data, and checkpoints are available at
https://qiushisun.github.io/OS-Genesis-Home/{OS-Genesis Homepage}.
67720d79b3163a95a653baaf
Qiushi Sun
claimed_verified
2025-01-02T10:19:16.620Z
6064a0eeb1703ddba0d458b9
https://cdn-avatars.huggingface.co/v1/production/uploads/1617207525789-noauth.png
Qiushi
user
QiushiSun
67720d79b3163a95a653bab0
Kanzhi Cheng
claimed_verified
2025-01-07T08:42:48.096Z
63340dbbd92c5842ae71d1e9
/avatars/3a3182996bd41b526dcbfa8687d91963.svg
Kanzhi Cheng
user
cckevinn
67720d79b3163a95a653bab1
Zichen Ding
claimed_verified
2025-01-02T10:19:18.679Z
642b9861bb77f8456634b048
https://cdn-avatars.huggingface.co/v1/production/uploads/642b9861bb77f8456634b048/ZT-oJrw5BsADC-gZT_i25.jpeg
Zichen Ding
user
heroding77
67720d79b3163a95a653bab2
Chuanyang Jin
claimed_verified
2025-01-03T14:05:28.148Z
64beb69801f1983a86a05de2
https://cdn-avatars.huggingface.co/v1/production/uploads/64beb69801f1983a86a05de2/tFyCoqZ6gT8NWkZfuncID.jpeg
Chuanyang Jin
user
Chuanyang-Jin
67720d79b3163a95a653bab3
Yian Wang
67720d79b3163a95a653bab4
Fangzhi Xu
claimed_verified
2025-01-03T14:05:20.429Z
64e6cf78ecce34cb442dc889
https://cdn-avatars.huggingface.co/v1/production/uploads/64e6cf78ecce34cb442dc889/qVZFiUEpBpSkmH8SQeinm.jpeg
Fangzhi Xu
user
xufangzhi
67720d79b3163a95a653bab5
Zhenyu Wu
67720d79b3163a95a653bab6
Chengyou Jia
claimed_verified
2025-01-03T14:05:30.296Z
6602548a68d519ed324b47c5
/avatars/5ab411f87440cc2a98c7a1c6a3ed5548.svg
ChengyouJia
user
ChengyouJia
67720d79b3163a95a653bab7
Liheng Chen
claimed_verified
2025-01-03T14:05:32.287Z
6561824484a9fbe322b9abc3
https://cdn-avatars.huggingface.co/v1/production/uploads/6561824484a9fbe322b9abc3/omS3D6PzGBD7kc4S7bOIO.png
LIHENG CHEN
user
Lemaqwq
67720d79b3163a95a653bab8
Zhoumianze Liu
67720d79b3163a95a653bab9
Ben Kao
67720d79b3163a95a653baba
Guohao Li
67720d79b3163a95a653babb
Junxian He
67720d79b3163a95a653babc
Yu Qiao
67720d79b3163a95a653babd
Zhiyong Wu
OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse
  Task Synthesis
Graphical User Interface (GUI) agents powered by Vision-Language Models
(VLMs) have demonstrated human-like computer control capability. Despite their
utility in advancing digital automation, a critical bottleneck persists:
collecting high-quality trajectory data for training. Common practices for
collecting such data rely on human supervision or synthetic data generation
through executing pre-defined tasks, which are either resource-intensive or
unable to guarantee data quality. Moreover, these methods suffer from limited
data diversity and significant gaps between synthetic data and real-world
environments. To address these challenges, we propose OS-Genesis, a novel GUI
data synthesis pipeline that reverses the conventional trajectory collection
process. Instead of relying on pre-defined tasks, OS-Genesis enables agents
first to perceive environments and perform step-wise interactions, then
retrospectively derive high-quality tasks to enable trajectory-level
exploration. A trajectory reward model is then employed to ensure the quality
of the generated trajectories. We demonstrate that training GUI agents with
OS-Genesis significantly improves their performance on highly challenging
online benchmarks. In-depth analysis further validates OS-Genesis's efficiency
and its superior data quality and diversity compared to existing synthesis
methods. Our codes, data, and checkpoints are available at
https://qiushisun.github.io/OS-Genesis-Home/{OS-Genesis Homepage}.
OS-Genesis is a novel GUI data synthesis pipeline that enhances the training of GUI agents by reversing the trajectory collection process to improve data quality and diversity.
Vision-Language Models GUI agents trajectory data human supervision synthetic data generation data diversity step-wise interactions trajectory reward model online benchmarks