title: Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
summary: Graphical User Interfaces (GUIs) are critical to human-computer interaction,
  yet automating GUI tasks remains challenging due to the complexity and
  variability of visual environments. Existing approaches often rely on textual
  representations of GUIs, which introduce limitations in generalization,
  efficiency, and scalability. In this paper, we introduce Aguvis, a unified pure
  vision-based framework for autonomous GUI agents that operates across various
  platforms. Our approach leverages image-based observations, and grounding
  instructions in natural language to visual elements, and employs a consistent
  action space to ensure cross-platform generalization. To address the
  limitations of previous work, we integrate explicit planning and reasoning
  within the model, enhancing its ability to autonomously navigate and interact
  with complex digital environments. We construct a large-scale dataset of GUI
  agent trajectories, incorporating multimodal reasoning and grounding, and
  employ a two-stage training pipeline that first focuses on general GUI
  grounding, followed by planning and reasoning. Through comprehensive
  experiments, we demonstrate that Aguvis surpasses previous state-of-the-art
  methods in both offline and real-world online scenarios, achieving, to our
  knowledge, the first fully autonomous pure vision GUI agent capable of
  performing tasks independently without collaboration with external
  closed-source models. We open-sourced all datasets, models, and training
  recipes to facilitate future research at https://aguvis-project.github.io/.
paper_authors._id: 67525cba61e5a1edbc685410
paper_authors.name: Yiheng Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T09:09:31.388Z
paper_authors._id: 601d29ab913ad3afd7b7ddb8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1620447944896-601d29ab913ad3afd7b7ddb8.jpeg
paper_authors.fullname: Yiheng Xu
paper_authors.type: user
paper_authors.user: ranpox
paper_authors._id: 67525cba61e5a1edbc685411
paper_authors.name: Zekun Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T10:37:55.632Z
paper_authors._id: 656832dfbd65fd41ee7aa8cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/656832dfbd65fd41ee7aa8cd/HHkyetTqNq1wIBPipzjQA.jpeg
paper_authors.fullname: Zekun Wang
paper_authors.type: user
paper_authors.user: kugwzk
paper_authors._id: 67525cba61e5a1edbc685412
paper_authors.name: Junli Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T09:09:27.583Z
paper_authors._id: 65f944d5056d465a38f49361
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/PWp43VsXeltiPifGxGwrn.jpeg
paper_authors.fullname: Junli Wang
paper_authors.type: user
paper_authors.user: ZeonLap
paper_authors._id: 67525cba61e5a1edbc685413
paper_authors.name: Dunjie Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T09:49:01.891Z
paper_authors._id: 669ca7e678115e16bdfc9bfc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/669ca7e678115e16bdfc9bfc/pku8NvQKqfNQACRqm1YrW.jpeg
paper_authors.fullname: Lu Dunjie
paper_authors.type: user
paper_authors.user: ludunjie
paper_authors._id: 67525cba61e5a1edbc685414
paper_authors.name: Tianbao Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T09:09:29.425Z
paper_authors._id: 618767e4238063b4615d042b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1636263880877-noauth.jpeg
paper_authors.fullname: Tianbao Xie
paper_authors.type: user
paper_authors.user: tianbaoxiexxx
paper_authors._id: 67525cba61e5a1edbc685415
paper_authors.name: Amrita Saha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T09:48:52.752Z
paper_authors._id: 6461c2905dba83471db3be53
paper_authors.avatarUrl: /avatars/6e36cf86201d590ac729a75d4a439cde.svg
paper_authors.fullname: Amrita Saha
paper_authors.type: user
paper_authors.user: amritasaha87
paper_authors._id: 67525cba61e5a1edbc685416
paper_authors.name: Doyen Sahoo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T09:48:40.670Z
paper_authors._id: 65f84fd980481173afd91233
paper_authors.avatarUrl: /avatars/6ac7bd6beba24d1476c5179b88c9e3fa.svg
paper_authors.fullname: Doyen
paper_authors.type: user
paper_authors.user: doyensahoo
paper_authors._id: 67525cba61e5a1edbc685417
paper_authors.name: Tao Yu
paper_authors._id: 67525cba61e5a1edbc685418
paper_authors.name: Caiming Xiong
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-12-06T02:09:01.442Z
paper_authors._id: 649dbcc4e0fff1ed099dc80a
paper_authors.avatarUrl: /avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
paper_authors.fullname: Caiming Xiong
paper_authors.type: user
paper_authors.user: cxiong
paper_title: Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
paper_summary: Graphical User Interfaces (GUIs) are critical to human-computer interaction,
  yet automating GUI tasks remains challenging due to the complexity and
  variability of visual environments. Existing approaches often rely on textual
  representations of GUIs, which introduce limitations in generalization,
  efficiency, and scalability. In this paper, we introduce Aguvis, a unified pure
  vision-based framework for autonomous GUI agents that operates across various
  platforms. Our approach leverages image-based observations, and grounding
  instructions in natural language to visual elements, and employs a consistent
  action space to ensure cross-platform generalization. To address the
  limitations of previous work, we integrate explicit planning and reasoning
  within the model, enhancing its ability to autonomously navigate and interact
  with complex digital environments. We construct a large-scale dataset of GUI
  agent trajectories, incorporating multimodal reasoning and grounding, and
  employ a two-stage training pipeline that first focuses on general GUI
  grounding, followed by planning and reasoning. Through comprehensive
  experiments, we demonstrate that Aguvis surpasses previous state-of-the-art
  methods in both offline and real-world online scenarios, achieving, to our
  knowledge, the first fully autonomous pure vision GUI agent capable of
  performing tasks independently without collaboration with external
  closed-source models. We open-sourced all datasets, models, and training
  recipes to facilitate future research at https://aguvis-project.github.io/.
paper_ai_keywords: image-based observations natural language to visual elements consistent action space explicit planning reasoning multimodal reasoning grounding two-stage training pipeline general GUI grounding GUI grounding