Breaking the Data Barrier -- Building GUI Agents Through Task
  Generalization
Graphical User Interface (GUI) agents offer cross-platform solutions for
automating complex digital tasks, with significant potential to transform
productivity workflows. However, their performance is often constrained by the
scarcity of high-quality trajectory data. To address this limitation, we
propose training Vision Language Models (VLMs) on data-rich,
reasoning-intensive tasks during a dedicated mid-training stage, and then
examine how incorporating these tasks facilitates generalization to GUI
planning scenarios. Specifically, we explore a range of tasks with readily
available instruction-tuning data, including GUI perception, multimodal
reasoning, and textual reasoning. Through extensive experiments across 11
mid-training tasks, we demonstrate that: (1) Task generalization proves highly
effective, yielding substantial improvements across most settings. For
instance, multimodal mathematical reasoning enhances performance on
AndroidWorld by an absolute 6.3%. Remarkably, text-only mathematical data
significantly boosts GUI web agent performance, achieving a 5.6% improvement on
WebArena and 5.4% improvement on AndroidWorld, underscoring notable cross-modal
generalization from text-based to visual domains; (2) Contrary to prior
assumptions, GUI perception data - previously considered closely aligned with
GUI agent tasks and widely utilized for training - has a comparatively limited
impact on final performance; (3) Building on these insights, we identify the
most effective mid-training tasks and curate optimized mixture datasets,
resulting in absolute performance gains of 8.0% on WebArena and 12.2% on
AndroidWorld. Our work provides valuable insights into cross-domain knowledge
transfer for GUI agents and offers a practical approach to addressing data
scarcity challenges in this emerging field. The code, data and models will be
available at https://github.com/hkust-nlp/GUIMid.
67fdf68d04d0302ef5ec0239
Junlei Zhang
claimed_verified
2025-04-15T07:52:43.667Z
63b76e716fc56e43c3c22ca8
/avatars/99c760e6a60e65d69d57a8b92a40f623.svg
Junlei Zhang
user
leoozy
67fdf68d04d0302ef5ec023a
Zichen Ding
admin_assigned
2025-04-15T08:29:38.552Z
642b9861bb77f8456634b048
https://cdn-avatars.huggingface.co/v1/production/uploads/642b9861bb77f8456634b048/ZT-oJrw5BsADC-gZT_i25.jpeg
Zichen Ding
user
heroding77
67fdf68d04d0302ef5ec023b
Chang Ma
admin_assigned
2025-04-15T08:29:44.813Z
637f22fd932a61b89aeeea37
/avatars/342957f8242d4edaf1d58e1274313afe.svg
Chang Ma
user
changma
67fdf68d04d0302ef5ec023c
Zijie Chen
67fdf68d04d0302ef5ec023d
Qiushi Sun
admin_assigned
2025-04-15T08:30:05.846Z
6064a0eeb1703ddba0d458b9
https://cdn-avatars.huggingface.co/v1/production/uploads/1617207525789-noauth.png
Qiushi
user
QiushiSun
67fdf68d04d0302ef5ec023e
Zhenzhong Lan
67fdf68d04d0302ef5ec023f
Junxian He
admin_assigned
2025-04-15T08:30:19.230Z
615f34ec3f6d24d67c1b5c78
/avatars/6dcff6477993d9e57c5cb92b6f95eb66.svg
Junxian He
user
jxhe
Breaking the Data Barrier -- Building GUI Agents Through Task
  Generalization
Graphical User Interface (GUI) agents offer cross-platform solutions for
automating complex digital tasks, with significant potential to transform
productivity workflows. However, their performance is often constrained by the
scarcity of high-quality trajectory data. To address this limitation, we
propose training Vision Language Models (VLMs) on data-rich,
reasoning-intensive tasks during a dedicated mid-training stage, and then
examine how incorporating these tasks facilitates generalization to GUI
planning scenarios. Specifically, we explore a range of tasks with readily
available instruction-tuning data, including GUI perception, multimodal
reasoning, and textual reasoning. Through extensive experiments across 11
mid-training tasks, we demonstrate that: (1) Task generalization proves highly
effective, yielding substantial improvements across most settings. For
instance, multimodal mathematical reasoning enhances performance on
AndroidWorld by an absolute 6.3%. Remarkably, text-only mathematical data
significantly boosts GUI web agent performance, achieving a 5.6% improvement on
WebArena and 5.4% improvement on AndroidWorld, underscoring notable cross-modal
generalization from text-based to visual domains; (2) Contrary to prior
assumptions, GUI perception data - previously considered closely aligned with
GUI agent tasks and widely utilized for training - has a comparatively limited
impact on final performance; (3) Building on these insights, we identify the
most effective mid-training tasks and curate optimized mixture datasets,
resulting in absolute performance gains of 8.0% on WebArena and 12.2% on
AndroidWorld. Our work provides valuable insights into cross-domain knowledge
transfer for GUI agents and offers a practical approach to addressing data
scarcity challenges in this emerging field. The code, data and models will be
available at https://github.com/hkust-nlp/GUIMid.
Training Vision Language Models on diverse mid-training tasks enhances generalization to GUI planning scenarios, significantly boosting performance through cross-modal generalization.
Vision Language Models VLMs GUI perception multimodal reasoning textual reasoning AndroidWorld WebArena GUI web agents cross-domain knowledge transfer GUI agents data scarcity