title: Large Language Model-Brained GUI Agents: A Survey
summary: GUIs have long been central to human-computer interaction, providing an
  intuitive and visually-driven way to access and interact with digital systems.
  The advent of LLMs, particularly multimodal models, has ushered in a new era of
  GUI automation. They have demonstrated exceptional capabilities in natural
  language understanding, code generation, and visual processing. This has paved
  the way for a new generation of LLM-brained GUI agents capable of interpreting
  complex GUI elements and autonomously executing actions based on natural
  language instructions. These agents represent a paradigm shift, enabling users
  to perform intricate, multi-step tasks through simple conversational commands.
  Their applications span across web navigation, mobile app interactions, and
  desktop automation, offering a transformative user experience that
  revolutionizes how individuals interact with software. This emerging field is
  rapidly advancing, with significant progress in both research and industry.
    To provide a structured understanding of this trend, this paper presents a
  comprehensive survey of LLM-brained GUI agents, exploring their historical
  evolution, core components, and advanced techniques. We address research
  questions such as existing GUI agent frameworks, the collection and utilization
  of data for training specialized GUI agents, the development of large action
  models tailored for GUI tasks, and the evaluation metrics and benchmarks
  necessary to assess their effectiveness. Additionally, we examine emerging
  applications powered by these agents. Through a detailed analysis, this survey
  identifies key research gaps and outlines a roadmap for future advancements in
  the field. By consolidating foundational knowledge and state-of-the-art
  developments, this work aims to guide both researchers and practitioners in
  overcoming challenges and unlocking the full potential of LLM-brained GUI
  agents.
paper_authors._id: 674803e9833196ed1932d215
paper_authors.name: Chaoyun Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T10:42:30.670Z
paper_authors._id: 654dbac9938fbf1e696be8aa
paper_authors.avatarUrl: /avatars/b3c4035c48169c1bfb04a439fce3499f.svg
paper_authors.fullname: Chaoyun Zhang
paper_authors.type: user
paper_authors.user: vyokky
paper_authors._id: 674803e9833196ed1932d216
paper_authors.name: Shilin He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T10:42:36.575Z
paper_authors._id: 62c6df026a092eda1f1ab6e5
paper_authors.avatarUrl: /avatars/d58fff1a157b189ce2617889ef5f6e2f.svg
paper_authors.fullname: Shilin He
paper_authors.type: user
paper_authors.user: shilhe
paper_authors._id: 674803e9833196ed1932d217
paper_authors.name: Jiaxu Qian
paper_authors._id: 674803e9833196ed1932d218
paper_authors.name: Bowen Li
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-01-10T09:30:59.562Z
paper_authors._id: 61cd553d3dd34ba1985e075b
paper_authors.avatarUrl: /avatars/5c39e4ac0892decc3af8e08109469196.svg
paper_authors.fullname: Bowen Li
paper_authors.type: user
paper_authors.user: bowenli
paper_authors._id: 674803e9833196ed1932d219
paper_authors.name: Liqun Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T09:19:23.018Z
paper_authors._id: 666933c97bf97e24f7b5266e
paper_authors.avatarUrl: /avatars/283961b37d463a386b08ad33dacca0f4.svg
paper_authors.fullname: Liqun Li
paper_authors.type: user
paper_authors.user: liqul
paper_authors._id: 674803e9833196ed1932d21a
paper_authors.name: Si Qin
paper_authors._id: 674803e9833196ed1932d21b
paper_authors.name: Yu Kang
paper_authors._id: 674803e9833196ed1932d21c
paper_authors.name: Minghua Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T10:42:51.136Z
paper_authors._id: 653b52098abd634b83f4fd38
paper_authors.avatarUrl: /avatars/c9c5023b030e94c1b1abfb7a1c1dfaf3.svg
paper_authors.fullname: MingHua Ma
paper_authors.type: user
paper_authors.user: Gezelligheid520
paper_authors._id: 674803e9833196ed1932d21d
paper_authors.name: Qingwei Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T10:43:11.603Z
paper_authors._id: 652fc9f39bc50a6c0e435224
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652fc9f39bc50a6c0e435224/70OBVDHHBsxG2giJ-E3_1.jpeg
paper_authors.fullname: Lin Qingwei
paper_authors.type: user
paper_authors.user: Eliblo1969
paper_authors._id: 674803e9833196ed1932d21e
paper_authors.name: Saravan Rajmohan
paper_authors._id: 674803e9833196ed1932d21f
paper_authors.name: Dongmei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T10:43:30.415Z
paper_authors._id: 66473d2c7abe6ad66e81a3dd
paper_authors.avatarUrl: /avatars/82f40244806c06ffeaa1c4265e9725ea.svg
paper_authors.fullname: ZHANGDONGMEI
paper_authors.type: user
paper_authors.user: ZDM6426
paper_authors._id: 674803e9833196ed1932d220
paper_authors.name: Qi Zhang
paper_title: Large Language Model-Brained GUI Agents: A Survey
paper_summary: GUIs have long been central to human-computer interaction, providing an
  intuitive and visually-driven way to access and interact with digital systems.
  The advent of LLMs, particularly multimodal models, has ushered in a new era of
  GUI automation. They have demonstrated exceptional capabilities in natural
  language understanding, code generation, and visual processing. This has paved
  the way for a new generation of LLM-brained GUI agents capable of interpreting
  complex GUI elements and autonomously executing actions based on natural
  language instructions. These agents represent a paradigm shift, enabling users
  to perform intricate, multi-step tasks through simple conversational commands.
  Their applications span across web navigation, mobile app interactions, and
  desktop automation, offering a transformative user experience that
  revolutionizes how individuals interact with software. This emerging field is
  rapidly advancing, with significant progress in both research and industry.
    To provide a structured understanding of this trend, this paper presents a
  comprehensive survey of LLM-brained GUI agents, exploring their historical
  evolution, core components, and advanced techniques. We address research
  questions such as existing GUI agent frameworks, the collection and utilization
  of data for training specialized GUI agents, the development of large action
  models tailored for GUI tasks, and the evaluation metrics and benchmarks
  necessary to assess their effectiveness. Additionally, we examine emerging
  applications powered by these agents. Through a detailed analysis, this survey
  identifies key research gaps and outlines a roadmap for future advancements in
  the field. By consolidating foundational knowledge and state-of-the-art
  developments, this work aims to guide both researchers and practitioners in
  overcoming challenges and unlocking the full potential of LLM-brained GUI
  agents.