A3: Android Agent Arena for Mobile GUI Agents
AI agents have become increasingly prevalent in recent years, driven by
significant advancements in the field of large language models (LLMs). Mobile
GUI agents, a subset of AI agents, are designed to autonomously perform tasks
on mobile devices. While numerous studies have introduced agents, datasets, and
benchmarks to advance mobile GUI agent research, many existing datasets focus
on static frame evaluations and fail to provide a comprehensive platform for
assessing performance on real-world, in-the-wild tasks. To address this gap, we
present Android Agent Arena (A3), a novel evaluation platform. Unlike existing
in-the-wild systems, A3 offers: (1) meaningful and practical tasks, such as
real-time online information retrieval and operational instructions; (2) a
larger, more flexible action space, enabling compatibility with agents trained
on any dataset; and (3) automated business-level LLM-based evaluation process.
A3 includes 21 widely used general third-party apps and 201 tasks
representative of common user scenarios, providing a robust foundation for
evaluating mobile GUI agents in real-world situations and a new autonomous
evaluation process for less human labor and coding expertise. The project is
available at https://yuxiangchai.github.io/Android-Agent-Arena/.
6777587d7fcef9e7b225cbae
Yuxiang Chai
claimed_verified
2025-01-03T14:03:27.243Z
6458ce236fa580137af5aa95
/avatars/db65a7332e375eb5daad5c1b076b1e3b.svg
Yuxiang Chai
user
Yuxiang007
6777587d7fcef9e7b225cbaf
Hanhao Li
6777587d7fcef9e7b225cbb0
Jiayu Zhang
6777587d7fcef9e7b225cbb1
Liang Liu
6777587d7fcef9e7b225cbb2
Guozhi Wang
6777587d7fcef9e7b225cbb3
Shuai Ren
6777587d7fcef9e7b225cbb4
Siyuan Huang
claimed_verified
2025-01-05T23:02:52.150Z
634e4120038b5879133552f5
/avatars/34ec861b4bbf1aecf927a7d6e726c7a4.svg
Siyuan
user
SiyuanH
6777587d7fcef9e7b225cbb5
Hongsheng Li
A3: Android Agent Arena for Mobile GUI Agents
AI agents have become increasingly prevalent in recent years, driven by
significant advancements in the field of large language models (LLMs). Mobile
GUI agents, a subset of AI agents, are designed to autonomously perform tasks
on mobile devices. While numerous studies have introduced agents, datasets, and
benchmarks to advance mobile GUI agent research, many existing datasets focus
on static frame evaluations and fail to provide a comprehensive platform for
assessing performance on real-world, in-the-wild tasks. To address this gap, we
present Android Agent Arena (A3), a novel evaluation platform. Unlike existing
in-the-wild systems, A3 offers: (1) meaningful and practical tasks, such as
real-time online information retrieval and operational instructions; (2) a
larger, more flexible action space, enabling compatibility with agents trained
on any dataset; and (3) automated business-level LLM-based evaluation process.
A3 includes 21 widely used general third-party apps and 201 tasks
representative of common user scenarios, providing a robust foundation for
evaluating mobile GUI agents in real-world situations and a new autonomous
evaluation process for less human labor and coding expertise. The project is
available at https://yuxiangchai.github.io/Android-Agent-Arena/.
Android Agent Arena (A3) provides an evaluation platform for mobile GUI agents with real-world tasks, flexible action spaces, and automated LLM-based testing.
large language models (LLMs) mobile GUI agents in-the-wild tasks action space autonomous evaluation real-time online information retrieval operational instructions