title: Android in the Wild: A Large-Scale Dataset for Android Device Control
summary: There is a growing interest in device-control systems that can interpret
  human natural language instructions and execute them on a digital device by
  directly controlling its user interface. We present a dataset for
  device-control research, Android in the Wild (AITW), which is orders of
  magnitude larger than current datasets. The dataset contains human
  demonstrations of device interactions, including the screens and actions, and
  corresponding natural language instructions. It consists of 715k episodes
  spanning 30k unique instructions, four versions of Android (v10-13),and eight
  device types (Pixel 2 XL to Pixel 6) with varying screen resolutions. It
  contains multi-step tasks that require semantic understanding of language and
  visual context. This dataset poses a new challenge: actions available through
  the user interface must be inferred from their visual appearance. And, instead
  of simple UI element-based actions, the action space consists of precise
  gestures (e.g., horizontal scrolls to operate carousel widgets). We organize
  our dataset to encourage robustness analysis of device-control systems, i.e.,
  how well a system performs in the presence of new task descriptions, new
  applications, or new platform versions. We develop two agents and report
  performance across the dataset. The dataset is available at
  https://github.com/google-research/google-research/tree/master/android_in_the_wild.
paper_authors._id: 64b8b23495bd42c7707d6e97
paper_authors.name: Christopher Rawles
paper_authors._id: 64b8b23495bd42c7707d6e98
paper_authors.name: Alice Li
paper_authors._id: 64b8b23495bd42c7707d6e99
paper_authors.name: Daniel Rodriguez
paper_authors._id: 64b8b23495bd42c7707d6e9a
paper_authors.name: Oriana Riva
paper_authors._id: 64b8b23495bd42c7707d6e9b
paper_authors.name: Timothy Lillicrap
paper_title: Android in the Wild: A Large-Scale Dataset for Android Device Control
paper_summary: There is a growing interest in device-control systems that can interpret
  human natural language instructions and execute them on a digital device by
  directly controlling its user interface. We present a dataset for
  device-control research, Android in the Wild (AITW), which is orders of
  magnitude larger than current datasets. The dataset contains human
  demonstrations of device interactions, including the screens and actions, and
  corresponding natural language instructions. It consists of 715k episodes
  spanning 30k unique instructions, four versions of Android (v10-13),and eight
  device types (Pixel 2 XL to Pixel 6) with varying screen resolutions. It
  contains multi-step tasks that require semantic understanding of language and
  visual context. This dataset poses a new challenge: actions available through
  the user interface must be inferred from their visual appearance. And, instead
  of simple UI element-based actions, the action space consists of precise
  gestures (e.g., horizontal scrolls to operate carousel widgets). We organize
  our dataset to encourage robustness analysis of device-control systems, i.e.,
  how well a system performs in the presence of new task descriptions, new
  applications, or new platform versions. We develop two agents and report
  performance across the dataset. The dataset is available at
  https://github.com/google-research/google-research/tree/master/android_in_the_wild.
paper_ai_keywords: semantic understanding visual context visual appearance gesture-based actions carousel widgets robustness analysis