title: GUI Agents: A Survey
summary: Graphical User Interface (GUI) agents, powered by Large Foundation Models,
  have emerged as a transformative approach to automating human-computer
  interaction. These agents autonomously interact with digital systems or
  software applications via GUIs, emulating human actions such as clicking,
  typing, and navigating visual elements across diverse platforms. Motivated by
  the growing interest and fundamental importance of GUI agents, we provide a
  comprehensive survey that categorizes their benchmarks, evaluation metrics,
  architectures, and training methods. We propose a unified framework that
  delineates their perception, reasoning, planning, and acting capabilities.
  Furthermore, we identify important open challenges and discuss key future
  directions. Finally, this work serves as a basis for practitioners and
  researchers to gain an intuitive understanding of current progress, techniques,
  benchmarks, and critical open problems that remain to be addressed.
paper_authors._id: 676389c7c998c60976a9a943
paper_authors.name: Dang Nguyen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-12-19T02:49:44.513Z
paper_authors._id: 64ba4c2565535cf237da429a
paper_authors.avatarUrl: /avatars/a8af0e748686e9d3364f1beaa5039ddb.svg
paper_authors.fullname: Dang Nguyen
paper_authors.type: user
paper_authors.user: dangmn
paper_authors._id: 676389c7c998c60976a9a944
paper_authors.name: Jian Chen
paper_authors._id: 676389c7c998c60976a9a945
paper_authors.name: Yu Wang
paper_authors._id: 676389c7c998c60976a9a946
paper_authors.name: Gang Wu
paper_authors._id: 676389c7c998c60976a9a947
paper_authors.name: Namyong Park
paper_authors._id: 676389c7c998c60976a9a948
paper_authors.name: Zhengmian Hu
paper_authors._id: 676389c7c998c60976a9a949
paper_authors.name: Hanjia Lyu
paper_authors._id: 676389c7c998c60976a9a94a
paper_authors.name: Junda Wu
paper_authors._id: 676389c7c998c60976a9a94b
paper_authors.name: Ryan Aponte
paper_authors._id: 676389c7c998c60976a9a94c
paper_authors.name: Yu Xia
paper_authors._id: 676389c7c998c60976a9a94d
paper_authors.name: Xintong Li
paper_authors._id: 676389c7c998c60976a9a94e
paper_authors.name: Jing Shi
paper_authors._id: 676389c7c998c60976a9a94f
paper_authors.name: Hongjie Chen
paper_authors._id: 676389c7c998c60976a9a950
paper_authors.name: Viet Dac Lai
paper_authors._id: 676389c7c998c60976a9a951
paper_authors.name: Zhouhang Xie
paper_authors._id: 676389c7c998c60976a9a952
paper_authors.name: Sungchul Kim
paper_authors._id: 676389c7c998c60976a9a953
paper_authors.name: Ruiyi Zhang
paper_authors._id: 676389c7c998c60976a9a954
paper_authors.name: Tong Yu
paper_authors._id: 676389c7c998c60976a9a955
paper_authors.name: Mehrab Tanjim
paper_authors._id: 676389c7c998c60976a9a956
paper_authors.name: Nesreen K. Ahmed
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T08:42:51.987Z
paper_authors._id: 663e588ccca86c8371a913d9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/663e588ccca86c8371a913d9/wgVGw5U77cqwtjovhsXyR.jpeg
paper_authors.fullname: Nesreen Ahmed
paper_authors.type: user
paper_authors.user: nkahmed
paper_authors._id: 676389c7c998c60976a9a957
paper_authors.name: Puneet Mathur
paper_authors._id: 676389c7c998c60976a9a958
paper_authors.name: Seunghyun Yoon
paper_authors._id: 676389c7c998c60976a9a959
paper_authors.name: Lina Yao
paper_authors._id: 676389c7c998c60976a9a95a
paper_authors.name: Branislav Kveton
paper_authors._id: 676389c7c998c60976a9a95b
paper_authors.name: Thien Huu Nguyen
paper_authors._id: 676389c7c998c60976a9a95c
paper_authors.name: Trung Bui
paper_authors._id: 676389c7c998c60976a9a95d
paper_authors.name: Tianyi Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:01:31.207Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_authors._id: 676389c7c998c60976a9a95e
paper_authors.name: Ryan A. Rossi
paper_authors._id: 676389c7c998c60976a9a95f
paper_authors.name: Franck Dernoncourt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-19T13:18:13.285Z
paper_authors._id: 62c5947524171688a9feb992
paper_authors.avatarUrl: /avatars/5a151713b9eae8dc566f5957acee3475.svg
paper_authors.fullname: Franck Dernoncourt
paper_authors.type: user
paper_authors.user: Franck-Dernoncourt
paper_title: GUI Agents: A Survey
paper_summary: Graphical User Interface (GUI) agents, powered by Large Foundation Models,
  have emerged as a transformative approach to automating human-computer
  interaction. These agents autonomously interact with digital systems or
  software applications via GUIs, emulating human actions such as clicking,
  typing, and navigating visual elements across diverse platforms. Motivated by
  the growing interest and fundamental importance of GUI agents, we provide a
  comprehensive survey that categorizes their benchmarks, evaluation metrics,
  architectures, and training methods. We propose a unified framework that
  delineates their perception, reasoning, planning, and acting capabilities.
  Furthermore, we identify important open challenges and discuss key future
  directions. Finally, this work serves as a basis for practitioners and
  researchers to gain an intuitive understanding of current progress, techniques,
  benchmarks, and critical open problems that remain to be addressed.
paper_ai_summary: A survey of graphical user interface agents powered by large foundation models, detailing benchmarks, metrics, architectures, training methods, and future challenges in automating human-computer interaction.
paper_ai_keywords: Large Foundation Models GUI agents human-computer interaction autonomous interaction digital systems software applications clicking typing navigating visual elements perception reasoning planning acting unified framework open challenges future directions