Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
Graphical User Interfaces (GUIs) are critical to human-computer interaction,
yet automating GUI tasks remains challenging due to the complexity and
variability of visual environments. Existing approaches often rely on textual
representations of GUIs, which introduce limitations in generalization,
efficiency, and scalability. In this paper, we introduce Aguvis, a unified pure
vision-based framework for autonomous GUI agents that operates across various
platforms. Our approach leverages image-based observations, and grounding
instructions in natural language to visual elements, and employs a consistent
action space to ensure cross-platform generalization. To address the
limitations of previous work, we integrate explicit planning and reasoning
within the model, enhancing its ability to autonomously navigate and interact
with complex digital environments. We construct a large-scale dataset of GUI
agent trajectories, incorporating multimodal reasoning and grounding, and
employ a two-stage training pipeline that first focuses on general GUI
grounding, followed by planning and reasoning. Through comprehensive
experiments, we demonstrate that Aguvis surpasses previous state-of-the-art
methods in both offline and real-world online scenarios, achieving, to our
knowledge, the first fully autonomous pure vision GUI agent capable of
performing tasks independently without collaboration with external
closed-source models. We open-sourced all datasets, models, and training
recipes to facilitate future research at https://aguvis-project.github.io/.
67525cba61e5a1edbc685410
Yiheng Xu
claimed_verified
2024-12-06T09:09:31.388Z
601d29ab913ad3afd7b7ddb8
https://cdn-avatars.huggingface.co/v1/production/uploads/1620447944896-601d29ab913ad3afd7b7ddb8.jpeg
Yiheng Xu
user
ranpox
67525cba61e5a1edbc685411
Zekun Wang
claimed_verified
2024-12-06T10:37:55.632Z
656832dfbd65fd41ee7aa8cd
https://cdn-avatars.huggingface.co/v1/production/uploads/656832dfbd65fd41ee7aa8cd/HHkyetTqNq1wIBPipzjQA.jpeg
Zekun Wang
user
kugwzk
67525cba61e5a1edbc685412
Junli Wang
claimed_verified
2024-12-06T09:09:27.583Z
65f944d5056d465a38f49361
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/PWp43VsXeltiPifGxGwrn.jpeg
Junli Wang
user
ZeonLap
67525cba61e5a1edbc685413
Dunjie Lu
admin_assigned
2024-12-06T09:49:01.891Z
669ca7e678115e16bdfc9bfc
https://cdn-avatars.huggingface.co/v1/production/uploads/669ca7e678115e16bdfc9bfc/pku8NvQKqfNQACRqm1YrW.jpeg
Lu Dunjie
user
ludunjie
67525cba61e5a1edbc685414
Tianbao Xie
claimed_verified
2024-12-06T09:09:29.425Z
618767e4238063b4615d042b
https://cdn-avatars.huggingface.co/v1/production/uploads/1636263880877-noauth.jpeg
Tianbao Xie
user
tianbaoxiexxx
67525cba61e5a1edbc685415
Amrita Saha
admin_assigned
2024-12-06T09:48:52.752Z
6461c2905dba83471db3be53
/avatars/6e36cf86201d590ac729a75d4a439cde.svg
Amrita Saha
user
amritasaha87
67525cba61e5a1edbc685416
Doyen Sahoo
admin_assigned
2024-12-06T09:48:40.670Z
65f84fd980481173afd91233
/avatars/6ac7bd6beba24d1476c5179b88c9e3fa.svg
Doyen
user
doyensahoo
67525cba61e5a1edbc685417
Tao Yu
67525cba61e5a1edbc685418
Caiming Xiong
extracted_pending
2024-12-06T02:09:01.442Z
649dbcc4e0fff1ed099dc80a
/avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
Caiming Xiong
user
cxiong
Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
Graphical User Interfaces (GUIs) are critical to human-computer interaction,
yet automating GUI tasks remains challenging due to the complexity and
variability of visual environments. Existing approaches often rely on textual
representations of GUIs, which introduce limitations in generalization,
efficiency, and scalability. In this paper, we introduce Aguvis, a unified pure
vision-based framework for autonomous GUI agents that operates across various
platforms. Our approach leverages image-based observations, and grounding
instructions in natural language to visual elements, and employs a consistent
action space to ensure cross-platform generalization. To address the
limitations of previous work, we integrate explicit planning and reasoning
within the model, enhancing its ability to autonomously navigate and interact
with complex digital environments. We construct a large-scale dataset of GUI
agent trajectories, incorporating multimodal reasoning and grounding, and
employ a two-stage training pipeline that first focuses on general GUI
grounding, followed by planning and reasoning. Through comprehensive
experiments, we demonstrate that Aguvis surpasses previous state-of-the-art
methods in both offline and real-world online scenarios, achieving, to our
knowledge, the first fully autonomous pure vision GUI agent capable of
performing tasks independently without collaboration with external
closed-source models. We open-sourced all datasets, models, and training
recipes to facilitate future research at https://aguvis-project.github.io/.
Aguvis is a unified, vision-based framework for autonomous GUI agents that integrates language grounding and explicit planning, enabling cross-platform generalization and outperforming existing methods.
GUIs image-based observations natural language grounding consistent action space explicit planning multimodal reasoning dataset two-stage training pipeline real-world online scenarios