Android in the Wild: A Large-Scale Dataset for Android Device Control
There is a growing interest in device-control systems that can interpret
human natural language instructions and execute them on a digital device by
directly controlling its user interface. We present a dataset for
device-control research, Android in the Wild (AITW), which is orders of
magnitude larger than current datasets. The dataset contains human
demonstrations of device interactions, including the screens and actions, and
corresponding natural language instructions. It consists of 715k episodes
spanning 30k unique instructions, four versions of Android (v10-13),and eight
device types (Pixel 2 XL to Pixel 6) with varying screen resolutions. It
contains multi-step tasks that require semantic understanding of language and
visual context. This dataset poses a new challenge: actions available through
the user interface must be inferred from their visual appearance. And, instead
of simple UI element-based actions, the action space consists of precise
gestures (e.g., horizontal scrolls to operate carousel widgets). We organize
our dataset to encourage robustness analysis of device-control systems, i.e.,
how well a system performs in the presence of new task descriptions, new
applications, or new platform versions. We develop two agents and report
performance across the dataset. The dataset is available at
https://github.com/google-research/google-research/tree/master/android_in_the_wild.
64b8b23495bd42c7707d6e97
Christopher Rawles
64b8b23495bd42c7707d6e98
Alice Li
64b8b23495bd42c7707d6e99
Daniel Rodriguez
64b8b23495bd42c7707d6e9a
Oriana Riva
64b8b23495bd42c7707d6e9b
Timothy Lillicrap
Android in the Wild: A Large-Scale Dataset for Android Device Control
There is a growing interest in device-control systems that can interpret
human natural language instructions and execute them on a digital device by
directly controlling its user interface. We present a dataset for
device-control research, Android in the Wild (AITW), which is orders of
magnitude larger than current datasets. The dataset contains human
demonstrations of device interactions, including the screens and actions, and
corresponding natural language instructions. It consists of 715k episodes
spanning 30k unique instructions, four versions of Android (v10-13),and eight
device types (Pixel 2 XL to Pixel 6) with varying screen resolutions. It
contains multi-step tasks that require semantic understanding of language and
visual context. This dataset poses a new challenge: actions available through
the user interface must be inferred from their visual appearance. And, instead
of simple UI element-based actions, the action space consists of precise
gestures (e.g., horizontal scrolls to operate carousel widgets). We organize
our dataset to encourage robustness analysis of device-control systems, i.e.,
how well a system performs in the presence of new task descriptions, new
applications, or new platform versions. We develop two agents and report
performance across the dataset. The dataset is available at
https://github.com/google-research/google-research/tree/master/android_in_the_wild.
A dataset named Android in the Wild challenges device-control systems with natural language instructions, requiring semantic understanding and inference of precise gestures from visual appearance.
device-control systems natural language instructions Android in the Wild (AITW) semantic understanding visual appearance multi-step tasks gesture inference