title: OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse
    Task Synthesis
summary: Graphical User Interface (GUI) agents powered by Vision-Language Models
  (VLMs) have demonstrated human-like computer control capability. Despite their
  utility in advancing digital automation, a critical bottleneck persists:
  collecting high-quality trajectory data for training. Common practices for
  collecting such data rely on human supervision or synthetic data generation
  through executing pre-defined tasks, which are either resource-intensive or
  unable to guarantee data quality. Moreover, these methods suffer from limited
  data diversity and significant gaps between synthetic data and real-world
  environments. To address these challenges, we propose OS-Genesis, a novel GUI
  data synthesis pipeline that reverses the conventional trajectory collection
  process. Instead of relying on pre-defined tasks, OS-Genesis enables agents
  first to perceive environments and perform step-wise interactions, then
  retrospectively derive high-quality tasks to enable trajectory-level
  exploration. A trajectory reward model is then employed to ensure the quality
  of the generated trajectories. We demonstrate that training GUI agents with
  OS-Genesis significantly improves their performance on highly challenging
  online benchmarks. In-depth analysis further validates OS-Genesis's efficiency
  and its superior data quality and diversity compared to existing synthesis
  methods. Our codes, data, and checkpoints are available at
  https://qiushisun.github.io/OS-Genesis-Home/{OS-Genesis Homepage}.
paper_authors._id: 67720d79b3163a95a653baaf
paper_authors.name: Qiushi Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-02T10:19:16.620Z
paper_authors._id: 6064a0eeb1703ddba0d458b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1617207525789-noauth.png
paper_authors.fullname: Qiushi
paper_authors.type: user
paper_authors.user: QiushiSun
paper_authors._id: 67720d79b3163a95a653bab0
paper_authors.name: Kanzhi Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T08:42:48.096Z
paper_authors._id: 63340dbbd92c5842ae71d1e9
paper_authors.avatarUrl: /avatars/3a3182996bd41b526dcbfa8687d91963.svg
paper_authors.fullname: Kanzhi Cheng
paper_authors.type: user
paper_authors.user: cckevinn
paper_authors._id: 67720d79b3163a95a653bab1
paper_authors.name: Zichen Ding
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-02T10:19:18.679Z
paper_authors._id: 642b9861bb77f8456634b048
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642b9861bb77f8456634b048/ZT-oJrw5BsADC-gZT_i25.jpeg
paper_authors.fullname: Zichen Ding
paper_authors.type: user
paper_authors.user: heroding77
paper_authors._id: 67720d79b3163a95a653bab2
paper_authors.name: Chuanyang Jin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:05:28.148Z
paper_authors._id: 64beb69801f1983a86a05de2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64beb69801f1983a86a05de2/tFyCoqZ6gT8NWkZfuncID.jpeg
paper_authors.fullname: Chuanyang Jin
paper_authors.type: user
paper_authors.user: Chuanyang-Jin
paper_authors._id: 67720d79b3163a95a653bab3
paper_authors.name: Yian Wang
paper_authors._id: 67720d79b3163a95a653bab4
paper_authors.name: Fangzhi Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:05:20.429Z
paper_authors._id: 64e6cf78ecce34cb442dc889
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e6cf78ecce34cb442dc889/qVZFiUEpBpSkmH8SQeinm.jpeg
paper_authors.fullname: Fangzhi Xu
paper_authors.type: user
paper_authors.user: xufangzhi
paper_authors._id: 67720d79b3163a95a653bab5
paper_authors.name: Zhenyu Wu
paper_authors._id: 67720d79b3163a95a653bab6
paper_authors.name: Chengyou Jia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:05:30.296Z
paper_authors._id: 6602548a68d519ed324b47c5
paper_authors.avatarUrl: /avatars/5ab411f87440cc2a98c7a1c6a3ed5548.svg
paper_authors.fullname: ChengyouJia
paper_authors.type: user
paper_authors.user: ChengyouJia
paper_authors._id: 67720d79b3163a95a653bab7
paper_authors.name: Liheng Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:05:32.287Z
paper_authors._id: 6561824484a9fbe322b9abc3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6561824484a9fbe322b9abc3/omS3D6PzGBD7kc4S7bOIO.png
paper_authors.fullname: LIHENG CHEN
paper_authors.type: user
paper_authors.user: Lemaqwq
paper_authors._id: 67720d79b3163a95a653bab8
paper_authors.name: Zhoumianze Liu
paper_authors._id: 67720d79b3163a95a653bab9
paper_authors.name: Ben Kao
paper_authors._id: 67720d79b3163a95a653baba
paper_authors.name: Guohao Li
paper_authors._id: 67720d79b3163a95a653babb
paper_authors.name: Junxian He
paper_authors._id: 67720d79b3163a95a653babc
paper_authors.name: Yu Qiao
paper_authors._id: 67720d79b3163a95a653babd
paper_authors.name: Zhiyong Wu
paper_title: OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse
    Task Synthesis
paper_summary: Graphical User Interface (GUI) agents powered by Vision-Language Models
  (VLMs) have demonstrated human-like computer control capability. Despite their
  utility in advancing digital automation, a critical bottleneck persists:
  collecting high-quality trajectory data for training. Common practices for
  collecting such data rely on human supervision or synthetic data generation
  through executing pre-defined tasks, which are either resource-intensive or
  unable to guarantee data quality. Moreover, these methods suffer from limited
  data diversity and significant gaps between synthetic data and real-world
  environments. To address these challenges, we propose OS-Genesis, a novel GUI
  data synthesis pipeline that reverses the conventional trajectory collection
  process. Instead of relying on pre-defined tasks, OS-Genesis enables agents
  first to perceive environments and perform step-wise interactions, then
  retrospectively derive high-quality tasks to enable trajectory-level
  exploration. A trajectory reward model is then employed to ensure the quality
  of the generated trajectories. We demonstrate that training GUI agents with
  OS-Genesis significantly improves their performance on highly challenging
  online benchmarks. In-depth analysis further validates OS-Genesis's efficiency
  and its superior data quality and diversity compared to existing synthesis
  methods. Our codes, data, and checkpoints are available at
  https://qiushisun.github.io/OS-Genesis-Home/{OS-Genesis Homepage}.
paper_ai_keywords: Vision-Language Models (VLMs) Trajectory data Trajectory-level exploration Trajectory reward model OS-Genesis GUI agents