Large Language Model-Brained GUI Agents: A Survey
GUIs have long been central to human-computer interaction, providing an
intuitive and visually-driven way to access and interact with digital systems.
The advent of LLMs, particularly multimodal models, has ushered in a new era of
GUI automation. They have demonstrated exceptional capabilities in natural
language understanding, code generation, and visual processing. This has paved
the way for a new generation of LLM-brained GUI agents capable of interpreting
complex GUI elements and autonomously executing actions based on natural
language instructions. These agents represent a paradigm shift, enabling users
to perform intricate, multi-step tasks through simple conversational commands.
Their applications span across web navigation, mobile app interactions, and
desktop automation, offering a transformative user experience that
revolutionizes how individuals interact with software. This emerging field is
rapidly advancing, with significant progress in both research and industry.
  To provide a structured understanding of this trend, this paper presents a
comprehensive survey of LLM-brained GUI agents, exploring their historical
evolution, core components, and advanced techniques. We address research
questions such as existing GUI agent frameworks, the collection and utilization
of data for training specialized GUI agents, the development of large action
models tailored for GUI tasks, and the evaluation metrics and benchmarks
necessary to assess their effectiveness. Additionally, we examine emerging
applications powered by these agents. Through a detailed analysis, this survey
identifies key research gaps and outlines a roadmap for future advancements in
the field. By consolidating foundational knowledge and state-of-the-art
developments, this work aims to guide both researchers and practitioners in
overcoming challenges and unlocking the full potential of LLM-brained GUI
agents.
674803e9833196ed1932d215
Chaoyun Zhang
admin_assigned
2024-11-28T10:42:30.670Z
654dbac9938fbf1e696be8aa
/avatars/b3c4035c48169c1bfb04a439fce3499f.svg
Chaoyun Zhang
user
vyokky
674803e9833196ed1932d216
Shilin He
admin_assigned
2024-11-28T10:42:36.575Z
62c6df026a092eda1f1ab6e5
/avatars/d58fff1a157b189ce2617889ef5f6e2f.svg
Shilin He
user
shilhe
674803e9833196ed1932d217
Jiaxu Qian
674803e9833196ed1932d218
Bowen Li
extracted_confirmed
2025-01-10T09:30:59.562Z
61cd553d3dd34ba1985e075b
/avatars/5c39e4ac0892decc3af8e08109469196.svg
Bowen Li
user
bowenli
674803e9833196ed1932d219
Liqun Li
claimed_verified
2024-11-28T09:19:23.018Z
666933c97bf97e24f7b5266e
/avatars/283961b37d463a386b08ad33dacca0f4.svg
Liqun Li
user
liqul
674803e9833196ed1932d21a
Si Qin
674803e9833196ed1932d21b
Yu Kang
674803e9833196ed1932d21c
Minghua Ma
admin_assigned
2024-11-28T10:42:51.136Z
653b52098abd634b83f4fd38
/avatars/c9c5023b030e94c1b1abfb7a1c1dfaf3.svg
MingHua Ma
user
Gezelligheid520
674803e9833196ed1932d21d
Qingwei Lin
admin_assigned
2024-11-28T10:43:11.603Z
652fc9f39bc50a6c0e435224
https://cdn-avatars.huggingface.co/v1/production/uploads/652fc9f39bc50a6c0e435224/70OBVDHHBsxG2giJ-E3_1.jpeg
Lin Qingwei
user
Eliblo1969
674803e9833196ed1932d21e
Saravan Rajmohan
674803e9833196ed1932d21f
Dongmei Zhang
admin_assigned
2024-11-28T10:43:30.415Z
66473d2c7abe6ad66e81a3dd
/avatars/82f40244806c06ffeaa1c4265e9725ea.svg
ZHANGDONGMEI
user
ZDM6426
674803e9833196ed1932d220
Qi Zhang
Large Language Model-Brained GUI Agents: A Survey
GUIs have long been central to human-computer interaction, providing an
intuitive and visually-driven way to access and interact with digital systems.
The advent of LLMs, particularly multimodal models, has ushered in a new era of
GUI automation. They have demonstrated exceptional capabilities in natural
language understanding, code generation, and visual processing. This has paved
the way for a new generation of LLM-brained GUI agents capable of interpreting
complex GUI elements and autonomously executing actions based on natural
language instructions. These agents represent a paradigm shift, enabling users
to perform intricate, multi-step tasks through simple conversational commands.
Their applications span across web navigation, mobile app interactions, and
desktop automation, offering a transformative user experience that
revolutionizes how individuals interact with software. This emerging field is
rapidly advancing, with significant progress in both research and industry.
  To provide a structured understanding of this trend, this paper presents a
comprehensive survey of LLM-brained GUI agents, exploring their historical
evolution, core components, and advanced techniques. We address research
questions such as existing GUI agent frameworks, the collection and utilization
of data for training specialized GUI agents, the development of large action
models tailored for GUI tasks, and the evaluation metrics and benchmarks
necessary to assess their effectiveness. Additionally, we examine emerging
applications powered by these agents. Through a detailed analysis, this survey
identifies key research gaps and outlines a roadmap for future advancements in
the field. By consolidating foundational knowledge and state-of-the-art
developments, this work aims to guide both researchers and practitioners in
overcoming challenges and unlocking the full potential of LLM-brained GUI
agents.
LLM-brained GUI agents enable sophisticated interactions through natural language commands, automating tasks across various platforms and enhancing user experiences.
LLMs multimodal models GUI automation natural language understanding code generation visual processing GUI agents web navigation mobile app interactions desktop automation GUI agent frameworks data collection large action models evaluation metrics benchmarks