title: AppAgent: Multimodal Agents as Smartphone Users
summary: Recent advancements in large language models (LLMs) have led to the creation
  of intelligent agents capable of performing complex tasks. This paper
  introduces a novel LLM-based multimodal agent framework designed to operate
  smartphone applications. Our framework enables the agent to operate smartphone
  applications through a simplified action space, mimicking human-like
  interactions such as tapping and swiping. This novel approach bypasses the need
  for system back-end access, thereby broadening its applicability across diverse
  apps. Central to our agent's functionality is its innovative learning method.
  The agent learns to navigate and use new apps either through autonomous
  exploration or by observing human demonstrations. This process generates a
  knowledge base that the agent refers to for executing complex tasks across
  different applications. To demonstrate the practicality of our agent, we
  conducted extensive testing over 50 tasks in 10 different applications,
  including social media, email, maps, shopping, and sophisticated image editing
  tools. The results affirm our agent's proficiency in handling a diverse array
  of high-level tasks.
paper_authors._id: 6585cc0d9747b6c745a14ef1
paper_authors.name: Chi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-26T10:37:33.698Z
paper_authors._id: 64196320ed725fef64419c2a
paper_authors.avatarUrl: /avatars/96feb22fb5e8931d6c9e0ea06148266f.svg
paper_authors.fullname: Chi Zhang
paper_authors.type: user
paper_authors.user: DrChiZhang
paper_authors._id: 65850d0069dfb7bfc5ef9dcc
paper_authors.name: Zhao Yang
paper_authors._id: 65850d0069dfb7bfc5ef9dcd
paper_authors.name: Jiaxuan Liu
paper_authors._id: 65850d0069dfb7bfc5ef9dce
paper_authors.name: Yucheng Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T10:58:07.217Z
paper_authors._id: 655c65c7ae75c25ae7e9f183
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/655c65c7ae75c25ae7e9f183/DuodzOHTig1ui_TJ6Sjfd.jpeg
paper_authors.fullname: Yucheng Han
paper_authors.type: user
paper_authors.user: listen2you002
paper_authors._id: 65850d0069dfb7bfc5ef9dcf
paper_authors.name: Xin Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-26T10:37:39.494Z
paper_authors._id: 63b543b2bda8d44adf2d7d0a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b543b2bda8d44adf2d7d0a/G_AzO8iA-WwEcxvJEKVQ0.jpeg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: ChrisChenXin
paper_authors._id: 65850d0069dfb7bfc5ef9dd0
paper_authors.name: Zebiao Huang
paper_authors._id: 65850d0069dfb7bfc5ef9dd1
paper_authors.name: Bin Fu
paper_authors._id: 65850d0069dfb7bfc5ef9dd2
paper_authors.name: Gang Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T10:59:08.698Z
paper_authors._id: 63417332c5565a4b8d43a0d8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63417332c5565a4b8d43a0d8/MmnYG7Wu2Z_lqCBvTfJmy.png
paper_authors.fullname: Gang Yu
paper_authors.type: user
paper_authors.user: skicy
paper_title: AppAgent: Multimodal Agents as Smartphone Users
paper_summary: Recent advancements in large language models (LLMs) have led to the creation
  of intelligent agents capable of performing complex tasks. This paper
  introduces a novel LLM-based multimodal agent framework designed to operate
  smartphone applications. Our framework enables the agent to operate smartphone
  applications through a simplified action space, mimicking human-like
  interactions such as tapping and swiping. This novel approach bypasses the need
  for system back-end access, thereby broadening its applicability across diverse
  apps. Central to our agent's functionality is its innovative learning method.
  The agent learns to navigate and use new apps either through autonomous
  exploration or by observing human demonstrations. This process generates a
  knowledge base that the agent refers to for executing complex tasks across
  different applications. To demonstrate the practicality of our agent, we
  conducted extensive testing over 50 tasks in 10 different applications,
  including social media, email, maps, shopping, and sophisticated image editing
  tools. The results affirm our agent's proficiency in handling a diverse array
  of high-level tasks.
paper_ai_summary: A novel LLM-based multimodal agent learns to operate smartphone apps through autonomous exploration or imitation, demonstrating proficiency across diverse tasks.
paper_ai_keywords: large language models multimodal agent autonomous exploration human demonstrations knowledge base high-level tasks