title: A3: Android Agent Arena for Mobile GUI Agents
summary: AI agents have become increasingly prevalent in recent years, driven by
  significant advancements in the field of large language models (LLMs). Mobile
  GUI agents, a subset of AI agents, are designed to autonomously perform tasks
  on mobile devices. While numerous studies have introduced agents, datasets, and
  benchmarks to advance mobile GUI agent research, many existing datasets focus
  on static frame evaluations and fail to provide a comprehensive platform for
  assessing performance on real-world, in-the-wild tasks. To address this gap, we
  present Android Agent Arena (A3), a novel evaluation platform. Unlike existing
  in-the-wild systems, A3 offers: (1) meaningful and practical tasks, such as
  real-time online information retrieval and operational instructions; (2) a
  larger, more flexible action space, enabling compatibility with agents trained
  on any dataset; and (3) automated business-level LLM-based evaluation process.
  A3 includes 21 widely used general third-party apps and 201 tasks
  representative of common user scenarios, providing a robust foundation for
  evaluating mobile GUI agents in real-world situations and a new autonomous
  evaluation process for less human labor and coding expertise. The project is
  available at https://yuxiangchai.github.io/Android-Agent-Arena/.
paper_authors._id: 6777587d7fcef9e7b225cbae
paper_authors.name: Yuxiang Chai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:03:27.243Z
paper_authors._id: 6458ce236fa580137af5aa95
paper_authors.avatarUrl: /avatars/db65a7332e375eb5daad5c1b076b1e3b.svg
paper_authors.fullname: Yuxiang Chai
paper_authors.type: user
paper_authors.user: Yuxiang007
paper_authors._id: 6777587d7fcef9e7b225cbaf
paper_authors.name: Hanhao Li
paper_authors._id: 6777587d7fcef9e7b225cbb0
paper_authors.name: Jiayu Zhang
paper_authors._id: 6777587d7fcef9e7b225cbb1
paper_authors.name: Liang Liu
paper_authors._id: 6777587d7fcef9e7b225cbb2
paper_authors.name: Guozhi Wang
paper_authors._id: 6777587d7fcef9e7b225cbb3
paper_authors.name: Shuai Ren
paper_authors._id: 6777587d7fcef9e7b225cbb4
paper_authors.name: Siyuan Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-05T23:02:52.150Z
paper_authors._id: 634e4120038b5879133552f5
paper_authors.avatarUrl: /avatars/34ec861b4bbf1aecf927a7d6e726c7a4.svg
paper_authors.fullname: Siyuan
paper_authors.type: user
paper_authors.user: SiyuanH
paper_authors._id: 6777587d7fcef9e7b225cbb5
paper_authors.name: Hongsheng Li
paper_title: A3: Android Agent Arena for Mobile GUI Agents
paper_summary: AI agents have become increasingly prevalent in recent years, driven by
  significant advancements in the field of large language models (LLMs). Mobile
  GUI agents, a subset of AI agents, are designed to autonomously perform tasks
  on mobile devices. While numerous studies have introduced agents, datasets, and
  benchmarks to advance mobile GUI agent research, many existing datasets focus
  on static frame evaluations and fail to provide a comprehensive platform for
  assessing performance on real-world, in-the-wild tasks. To address this gap, we
  present Android Agent Arena (A3), a novel evaluation platform. Unlike existing
  in-the-wild systems, A3 offers: (1) meaningful and practical tasks, such as
  real-time online information retrieval and operational instructions; (2) a
  larger, more flexible action space, enabling compatibility with agents trained
  on any dataset; and (3) automated business-level LLM-based evaluation process.
  A3 includes 21 widely used general third-party apps and 201 tasks
  representative of common user scenarios, providing a robust foundation for
  evaluating mobile GUI agents in real-world situations and a new autonomous
  evaluation process for less human labor and coding expertise. The project is
  available at https://yuxiangchai.github.io/Android-Agent-Arena/.
paper_ai_keywords: Text-at past large language models (LLMs) Mobile GUI agents Android Agent Arena (A3) Real-time online information LLM-based evaluation process feature-level evaluation