title: AndroidLab: Training and Systematic Benchmarking of Android Autonomous
    Agents
summary: Autonomous agents have become increasingly important for interacting with the
  real world. Android agents, in particular, have been recently a
  frequently-mentioned interaction method. However, existing studies for training
  and evaluating Android agents lack systematic research on both open-source and
  closed-source models. In this work, we propose AndroidLab as a systematic
  Android agent framework. It includes an operation environment with different
  modalities, action space, and a reproducible benchmark. It supports both large
  language models (LLMs) and multimodal models (LMMs) in the same action space.
  AndroidLab benchmark includes predefined Android virtual devices and 138 tasks
  across nine apps built on these devices. By using the AndroidLab environment,
  we develop an Android Instruction dataset and train six open-source LLMs and
  LMMs, lifting the average success rates from 4.59% to 21.50% for LLMs and from
  1.93% to 13.28% for LMMs. AndroidLab is open-sourced and publicly available at
  https://github.com/THUDM/Android-Lab.
paper_authors._id: 67299eee6cc63cc1ba362e9b
paper_authors.name: Yifan Xu
paper_authors._id: 67299eee6cc63cc1ba362e9c
paper_authors.name: Xiao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T16:09:19.323Z
paper_authors._id: 62f098602ca4d32a7cd87aba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1659934844914-62f098602ca4d32a7cd87aba.jpeg
paper_authors.fullname: Xiao Liu
paper_authors.type: user
paper_authors.user: ShawLiu
paper_authors._id: 67299eee6cc63cc1ba362e9d
paper_authors.name: Xueqiao Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T13:38:12.414Z
paper_authors._id: 6729afc3c519b000f45d053b
paper_authors.avatarUrl: /avatars/1d47cfd495e72316420fc16f7286df2b.svg
paper_authors.fullname: Xueqiao Sun
paper_authors.type: user
paper_authors.user: snow1007
paper_authors._id: 67299eee6cc63cc1ba362e9e
paper_authors.name: Siyi Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T13:38:51.528Z
paper_authors._id: 654cf05258c9d74272181118
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Dfx-hp3RLkP95gQtOM7T0.png
paper_authors.fullname: Siyi Cheng
paper_authors.type: user
paper_authors.user: LeakyKayO
paper_authors._id: 67299eee6cc63cc1ba362e9f
paper_authors.name: Hao Yu
paper_authors._id: 67299eee6cc63cc1ba362ea0
paper_authors.name: Hanyu Lai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T13:38:57.325Z
paper_authors._id: 62ea2d48b493272c269e8f34
paper_authors.avatarUrl: /avatars/89b215dafa503b51ab212a9b63c82aca.svg
paper_authors.fullname: Hanyu Lai
paper_authors.type: user
paper_authors.user: hanyullai
paper_authors._id: 67299eee6cc63cc1ba362ea1
paper_authors.name: Shudan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T13:39:04.042Z
paper_authors._id: 637883f3736e2989331da910
paper_authors.avatarUrl: /avatars/f72b579724d2d43889907e8b732e79e6.svg
paper_authors.fullname: Shudan Zhang
paper_authors.type: user
paper_authors.user: zdaniel0222
paper_authors._id: 67299eee6cc63cc1ba362ea2
paper_authors.name: Dan Zhang
paper_authors._id: 67299eee6cc63cc1ba362ea3
paper_authors.name: Jie Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T13:37:40.424Z
paper_authors._id: 640dff05474aa6f89556677e
paper_authors.avatarUrl: /avatars/1b4591c7322d649c797b3125148f1915.svg
paper_authors.fullname: Jie Tang
paper_authors.type: user
paper_authors.user: jerytang
paper_authors._id: 67299eee6cc63cc1ba362ea4
paper_authors.name: Yuxiao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T13:36:14.681Z
paper_authors._id: 640e73bdfdeaae1390857b62
paper_authors.avatarUrl: /avatars/cd6779e30f716002a7838ed93d5c0754.svg
paper_authors.fullname: Yuxiao Dong
paper_authors.type: user
paper_authors.user: yuxiaod
paper_title: AndroidLab: Training and Systematic Benchmarking of Android Autonomous
    Agents
paper_summary: Autonomous agents have become increasingly important for interacting with the
  real world. Android agents, in particular, have been recently a
  frequently-mentioned interaction method. However, existing studies for training
  and evaluating Android agents lack systematic research on both open-source and
  closed-source models. In this work, we propose AndroidLab as a systematic
  Android agent framework. It includes an operation environment with different
  modalities, action space, and a reproducible benchmark. It supports both large
  language models (LLMs) and multimodal models (LMMs) in the same action space.
  AndroidLab benchmark includes predefined Android virtual devices and 138 tasks
  across nine apps built on these devices. By using the AndroidLab environment,
  we develop an Android Instruction dataset and train six open-source LLMs and
  LMMs, lifting the average success rates from 4.59% to 21.50% for LLMs and from
  1.93% to 13.28% for LMMs. AndroidLab is open-sourced and publicly available at
  https://github.com/THUDM/Android-Lab.
paper_ai_keywords: Android agents AndroidLab operation environment modalities action space reproducible benchmark large language models (LLMs) multimodal models (LMMs) Android virtual devices Android Instruction dataset