AppAgentX: Evolving GUI Agents as Proficient Smartphone Users
Recent advancements in Large Language Models (LLMs) have led to the
development of intelligent LLM-based agents capable of interacting with
graphical user interfaces (GUIs). These agents demonstrate strong reasoning and
adaptability, enabling them to perform complex tasks that traditionally
required predefined rules. However, the reliance on step-by-step reasoning in
LLM-based agents often results in inefficiencies, particularly for routine
tasks. In contrast, traditional rule-based systems excel in efficiency but lack
the intelligence and flexibility to adapt to novel scenarios. To address this
challenge, we propose a novel evolutionary framework for GUI agents that
enhances operational efficiency while retaining intelligence and flexibility.
Our approach incorporates a memory mechanism that records the agent's task
execution history. By analyzing this history, the agent identifies repetitive
action sequences and evolves high-level actions that act as shortcuts,
replacing these low-level operations and improving efficiency. This allows the
agent to focus on tasks requiring more complex reasoning, while simplifying
routine actions. Experimental results on multiple benchmark tasks demonstrate
that our approach significantly outperforms existing methods in both efficiency
and accuracy. The code will be open-sourced to support further research.
67c7ebafdf15f5978ac987c3
Wenjia Jiang
67c7ebafdf15f5978ac987c4
Yangyang Zhuang
admin_assigned
2025-03-05T11:38:30.615Z
673afd8879b5a48d37f38000
https://cdn-avatars.huggingface.co/v1/production/uploads/673afd8879b5a48d37f38000/3-5-L6St2hw6MkH3uIOKA.jpeg
Yangyang Zhuang
user
yyzhuang2
67c7ebafdf15f5978ac987c5
Chenxi Song
admin_assigned
2025-03-05T11:38:35.946Z
65cf975bf5a15aa42146b208
/avatars/d62e3907331a24bb899b7a1fdf4299e7.svg
Song
user
ChenxiSong
67c7ebafdf15f5978ac987c6
Xu Yang
admin_assigned
2025-03-05T11:38:43.389Z
63104777cc8ed75decbd9500
/avatars/f9002d8bffa7304296718321d4c6f58a.svg
xuyang
user
xuyang
67c7ebafdf15f5978ac987c7
Chi Zhang
AppAgentX: Evolving GUI Agents as Proficient Smartphone Users
Recent advancements in Large Language Models (LLMs) have led to the
development of intelligent LLM-based agents capable of interacting with
graphical user interfaces (GUIs). These agents demonstrate strong reasoning and
adaptability, enabling them to perform complex tasks that traditionally
required predefined rules. However, the reliance on step-by-step reasoning in
LLM-based agents often results in inefficiencies, particularly for routine
tasks. In contrast, traditional rule-based systems excel in efficiency but lack
the intelligence and flexibility to adapt to novel scenarios. To address this
challenge, we propose a novel evolutionary framework for GUI agents that
enhances operational efficiency while retaining intelligence and flexibility.
Our approach incorporates a memory mechanism that records the agent's task
execution history. By analyzing this history, the agent identifies repetitive
action sequences and evolves high-level actions that act as shortcuts,
replacing these low-level operations and improving efficiency. This allows the
agent to focus on tasks requiring more complex reasoning, while simplifying
routine actions. Experimental results on multiple benchmark tasks demonstrate
that our approach significantly outperforms existing methods in both efficiency
and accuracy. The code will be open-sourced to support further research.
An evolutionary framework with memory for GUI agents improves efficiency by evolving high-level actions from task execution history, outperforming existing methods in efficiency and accuracy.
Large Language Models (LLMs) GUI agents step-by-step reasoning memory mechanism high-level actions low-level operations evolutionary framework