title: Breaking the Data Barrier -- Building GUI Agents Through Task
    Generalization
summary: Graphical User Interface (GUI) agents offer cross-platform solutions for
  automating complex digital tasks, with significant potential to transform
  productivity workflows. However, their performance is often constrained by the
  scarcity of high-quality trajectory data. To address this limitation, we
  propose training Vision Language Models (VLMs) on data-rich,
  reasoning-intensive tasks during a dedicated mid-training stage, and then
  examine how incorporating these tasks facilitates generalization to GUI
  planning scenarios. Specifically, we explore a range of tasks with readily
  available instruction-tuning data, including GUI perception, multimodal
  reasoning, and textual reasoning. Through extensive experiments across 11
  mid-training tasks, we demonstrate that: (1) Task generalization proves highly
  effective, yielding substantial improvements across most settings. For
  instance, multimodal mathematical reasoning enhances performance on
  AndroidWorld by an absolute 6.3%. Remarkably, text-only mathematical data
  significantly boosts GUI web agent performance, achieving a 5.6% improvement on
  WebArena and 5.4% improvement on AndroidWorld, underscoring notable cross-modal
  generalization from text-based to visual domains; (2) Contrary to prior
  assumptions, GUI perception data - previously considered closely aligned with
  GUI agent tasks and widely utilized for training - has a comparatively limited
  impact on final performance; (3) Building on these insights, we identify the
  most effective mid-training tasks and curate optimized mixture datasets,
  resulting in absolute performance gains of 8.0% on WebArena and 12.2% on
  AndroidWorld. Our work provides valuable insights into cross-domain knowledge
  transfer for GUI agents and offers a practical approach to addressing data
  scarcity challenges in this emerging field. The code, data and models will be
  available at https://github.com/hkust-nlp/GUIMid.
paper_authors._id: 67fdf68d04d0302ef5ec0239
paper_authors.name: Junlei Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T07:52:43.667Z
paper_authors._id: 63b76e716fc56e43c3c22ca8
paper_authors.avatarUrl: /avatars/99c760e6a60e65d69d57a8b92a40f623.svg
paper_authors.fullname: Junlei Zhang
paper_authors.type: user
paper_authors.user: leoozy
paper_authors._id: 67fdf68d04d0302ef5ec023a
paper_authors.name: Zichen Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:29:38.552Z
paper_authors._id: 642b9861bb77f8456634b048
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642b9861bb77f8456634b048/ZT-oJrw5BsADC-gZT_i25.jpeg
paper_authors.fullname: Zichen Ding
paper_authors.type: user
paper_authors.user: heroding77
paper_authors._id: 67fdf68d04d0302ef5ec023b
paper_authors.name: Chang Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:29:44.813Z
paper_authors._id: 637f22fd932a61b89aeeea37
paper_authors.avatarUrl: /avatars/342957f8242d4edaf1d58e1274313afe.svg
paper_authors.fullname: Chang Ma
paper_authors.type: user
paper_authors.user: changma
paper_authors._id: 67fdf68d04d0302ef5ec023c
paper_authors.name: Zijie Chen
paper_authors._id: 67fdf68d04d0302ef5ec023d
paper_authors.name: Qiushi Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:30:05.846Z
paper_authors._id: 6064a0eeb1703ddba0d458b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1617207525789-noauth.png
paper_authors.fullname: Qiushi
paper_authors.type: user
paper_authors.user: QiushiSun
paper_authors._id: 67fdf68d04d0302ef5ec023e
paper_authors.name: Zhenzhong Lan
paper_authors._id: 67fdf68d04d0302ef5ec023f
paper_authors.name: Junxian He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:30:19.230Z
paper_authors._id: 615f34ec3f6d24d67c1b5c78
paper_authors.avatarUrl: /avatars/6dcff6477993d9e57c5cb92b6f95eb66.svg
paper_authors.fullname: Junxian He
paper_authors.type: user
paper_authors.user: jxhe
paper_title: Breaking the Data Barrier -- Building GUI Agents Through Task
    Generalization
paper_summary: Graphical User Interface (GUI) agents offer cross-platform solutions for
  automating complex digital tasks, with significant potential to transform
  productivity workflows. However, their performance is often constrained by the
  scarcity of high-quality trajectory data. To address this limitation, we
  propose training Vision Language Models (VLMs) on data-rich,
  reasoning-intensive tasks during a dedicated mid-training stage, and then
  examine how incorporating these tasks facilitates generalization to GUI
  planning scenarios. Specifically, we explore a range of tasks with readily
  available instruction-tuning data, including GUI perception, multimodal
  reasoning, and textual reasoning. Through extensive experiments across 11
  mid-training tasks, we demonstrate that: (1) Task generalization proves highly
  effective, yielding substantial improvements across most settings. For
  instance, multimodal mathematical reasoning enhances performance on
  AndroidWorld by an absolute 6.3%. Remarkably, text-only mathematical data
  significantly boosts GUI web agent performance, achieving a 5.6% improvement on
  WebArena and 5.4% improvement on AndroidWorld, underscoring notable cross-modal
  generalization from text-based to visual domains; (2) Contrary to prior
  assumptions, GUI perception data - previously considered closely aligned with
  GUI agent tasks and widely utilized for training - has a comparatively limited
  impact on final performance; (3) Building on these insights, we identify the
  most effective mid-training tasks and curate optimized mixture datasets,
  resulting in absolute performance gains of 8.0% on WebArena and 12.2% on
  AndroidWorld. Our work provides valuable insights into cross-domain knowledge
  transfer for GUI agents and offers a practical approach to addressing data
  scarcity challenges in this emerging field. The code, data and models will be
  available at https://github.com/hkust-nlp/GUIMid.
paper_ai_summary: Training Vision Language Models on diverse mid-training tasks enhances generalization to GUI planning scenarios, significantly boosting performance through cross-modal generalization.
paper_ai_keywords: Vision Language Models VLMs GUI perception multimodal reasoning textual reasoning AndroidWorld WebArena GUI web agents cross-domain knowledge transfer GUI agents data scarcity