GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone
  GUI Navigation
We present MM-Navigator, a GPT-4V-based agent for the smartphone graphical
user interface (GUI) navigation task. MM-Navigator can interact with a
smartphone screen as human users, and determine subsequent actions to fulfill
given instructions. Our findings demonstrate that large multimodal models
(LMMs), specifically GPT-4V, excel in zero-shot GUI navigation through its
advanced screen interpretation, action reasoning, and precise action
localization capabilities. We first benchmark MM-Navigator on our collected iOS
screen dataset. According to human assessments, the system exhibited a 91\%
accuracy rate in generating reasonable action descriptions and a 75\% accuracy
rate in executing the correct actions for single-step instructions on iOS.
Additionally, we evaluate the model on a subset of an Android screen navigation
dataset, where the model outperforms previous GUI navigators in a zero-shot
fashion. Our benchmark and detailed analyses aim to lay a robust groundwork for
future research into the GUI navigation task. The project page is at
https://github.com/zzxslp/MM-Navigator.
6552e9bcb2d47c6675c91bbc
An Yan
claimed_verified
2024-04-22T06:55:46.871Z
634dfb9bbe5a827d48749f39
/avatars/a8df0eeb1df39f3935dd686a94768f30.svg
An Yan
user
zzxslp
6552e9bcb2d47c6675c91bbd
Zhengyuan Yang
admin_assigned
2023-11-14T10:34:14.297Z
630713411801ecc7d2592a7c
/avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
Zhengyuan Yang
user
zyang39
6552e9bcb2d47c6675c91bbe
Wanrong Zhu
claimed_verified
2023-11-14T09:36:54.022Z
633a7e91c8eb7b089034fb3c
/avatars/0e03d80aa469de131cc17bb0c5c016b9.svg
Wanrong Zhu
user
VegB
6552e9bcb2d47c6675c91bbf
Kevin Lin
claimed_verified
2023-11-14T09:36:50.637Z
6298fd95b58e71e2ac9f3ad8
/avatars/7d34644d537bc5c17cf1e4ce4095355c.svg
Kevin Lin
user
kevinlin311tw
6552e9bcb2d47c6675c91bc0
Linjie Li
admin_assigned
2023-11-14T10:34:30.187Z
63db16fff03c3d71ef397206
/avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
Linjie Li
user
linjieli222
6552e9bcb2d47c6675c91bc1
Jianfeng Wang
admin_assigned
2023-11-14T10:35:00.082Z
641404e74e5305c14f269b3c
/avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
Jianfeng Wang
user
amsword8
6552e9bcb2d47c6675c91bc2
Jianwei Yang
admin_assigned
2023-11-14T10:35:18.221Z
6125df7f25027fb1ea9c7a41
https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
Jianwei Yang
user
jw2yang
6552e9bcb2d47c6675c91bc3
Yiwu Zhong
extracted_confirmed
2023-11-14T04:43:45.018Z
62b4fe7ab25cb80fcf2ffd66
/avatars/22f6a05cdbf4224d29ec9259c9fdd7a4.svg
Yiwu Zhong
user
YiwuZhong
6552e9bcb2d47c6675c91bc4
Julian McAuley
6552e9bcb2d47c6675c91bc5
Jianfeng Gao
admin_assigned
2023-11-14T10:35:32.890Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
6552e9bcb2d47c6675c91bc6
Zicheng Liu
6552e9bcb2d47c6675c91bc7
Lijuan Wang
admin_assigned
2023-11-14T10:35:48.322Z
6413521d4e5305c14f22e110
/avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
Lijuan Wang
user
Lijuan
GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone
  GUI Navigation
We present MM-Navigator, a GPT-4V-based agent for the smartphone graphical
user interface (GUI) navigation task. MM-Navigator can interact with a
smartphone screen as human users, and determine subsequent actions to fulfill
given instructions. Our findings demonstrate that large multimodal models
(LMMs), specifically GPT-4V, excel in zero-shot GUI navigation through its
advanced screen interpretation, action reasoning, and precise action
localization capabilities. We first benchmark MM-Navigator on our collected iOS
screen dataset. According to human assessments, the system exhibited a 91\%
accuracy rate in generating reasonable action descriptions and a 75\% accuracy
rate in executing the correct actions for single-step instructions on iOS.
Additionally, we evaluate the model on a subset of an Android screen navigation
dataset, where the model outperforms previous GUI navigators in a zero-shot
fashion. Our benchmark and detailed analyses aim to lay a robust groundwork for
future research into the GUI navigation task. The project page is at
https://github.com/zzxslp/MM-Navigator.
A GPT-4V-based agent demonstrates high accuracy in zero-shot smartphone GUI navigation through advanced screen interpretation, action reasoning, and precise localization.
GPT-4V GUI navigation screen interpretation action reasoning precise action localization multimodal models zero-shot performance iOS Android