title: AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
summary: AI agents have drawn increasing attention mostly on their ability to perceive
  environments, understand tasks, and autonomously achieve goals. To advance
  research on AI agents in mobile scenarios, we introduce the Android
  Multi-annotation EXpo (AMEX), a comprehensive, large-scale dataset designed for
  generalist mobile GUI-control agents. Their capabilities of completing complex
  tasks by directly interacting with the graphical user interface (GUI) on mobile
  devices are trained and evaluated with the proposed dataset. AMEX comprises
  over 104K high-resolution screenshots from 110 popular mobile applications,
  which are annotated at multiple levels. Unlike existing mobile device-control
  datasets, e.g., MoTIF, AitW, etc., AMEX includes three levels of annotations:
  GUI interactive element grounding, GUI screen and element functionality
  descriptions, and complex natural language instructions, each averaging 13
  steps with stepwise GUI-action chains. We develop this dataset from a more
  instructive and detailed perspective, complementing the general settings of
  existing datasets. Additionally, we develop a baseline model SPHINX Agent and
  compare its performance across state-of-the-art agents trained on other
  datasets. To facilitate further research, we open-source our dataset, models,
  and relevant evaluation tools. The project is available at
  https://yuxiangchai.github.io/AMEX/
paper_authors._id: 66a30b583c7f0decff91b8a6
paper_authors.name: Yuxiang Chai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-26T07:40:16.403Z
paper_authors._id: 6458ce236fa580137af5aa95
paper_authors.avatarUrl: /avatars/db65a7332e375eb5daad5c1b076b1e3b.svg
paper_authors.fullname: Yuxiang Chai
paper_authors.type: user
paper_authors.user: Yuxiang007
paper_authors._id: 66a30b583c7f0decff91b8a7
paper_authors.name: Siyuan Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-17T09:31:20.873Z
paper_authors._id: 634e4120038b5879133552f5
paper_authors.avatarUrl: /avatars/34ec861b4bbf1aecf927a7d6e726c7a4.svg
paper_authors.fullname: Siyuan
paper_authors.type: user
paper_authors.user: SiyuanH
paper_authors._id: 66a30b583c7f0decff91b8a8
paper_authors.name: Yazhe Niu
paper_authors._id: 66a30b583c7f0decff91b8a9
paper_authors.name: Han Xiao
paper_authors._id: 66a30b583c7f0decff91b8aa
paper_authors.name: Liang Liu
paper_authors._id: 66a30b583c7f0decff91b8ab
paper_authors.name: Dingyu Zhang
paper_authors._id: 66a30b583c7f0decff91b8ac
paper_authors.name: Peng Gao
paper_authors._id: 66a30b583c7f0decff91b8ad
paper_authors.name: Shuai Ren
paper_authors._id: 66a30b583c7f0decff91b8ae
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-26T07:48:40.057Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_title: AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
paper_summary: AI agents have drawn increasing attention mostly on their ability to perceive
  environments, understand tasks, and autonomously achieve goals. To advance
  research on AI agents in mobile scenarios, we introduce the Android
  Multi-annotation EXpo (AMEX), a comprehensive, large-scale dataset designed for
  generalist mobile GUI-control agents. Their capabilities of completing complex
  tasks by directly interacting with the graphical user interface (GUI) on mobile
  devices are trained and evaluated with the proposed dataset. AMEX comprises
  over 104K high-resolution screenshots from 110 popular mobile applications,
  which are annotated at multiple levels. Unlike existing mobile device-control
  datasets, e.g., MoTIF, AitW, etc., AMEX includes three levels of annotations:
  GUI interactive element grounding, GUI screen and element functionality
  descriptions, and complex natural language instructions, each averaging 13
  steps with stepwise GUI-action chains. We develop this dataset from a more
  instructive and detailed perspective, complementing the general settings of
  existing datasets. Additionally, we develop a baseline model SPHINX Agent and
  compare its performance across state-of-the-art agents trained on other
  datasets. To facilitate further research, we open-source our dataset, models,
  and relevant evaluation tools. The project is available at
  https://yuxiangchai.github.io/AMEX/
paper_ai_keywords: GUI interactive element grounding GUI screen and element functionality descriptions complex natural language instructions SPHINX Agent