title: GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone
    GUI Navigation
summary: We present MM-Navigator, a GPT-4V-based agent for the smartphone graphical
  user interface (GUI) navigation task. MM-Navigator can interact with a
  smartphone screen as human users, and determine subsequent actions to fulfill
  given instructions. Our findings demonstrate that large multimodal models
  (LMMs), specifically GPT-4V, excel in zero-shot GUI navigation through its
  advanced screen interpretation, action reasoning, and precise action
  localization capabilities. We first benchmark MM-Navigator on our collected iOS
  screen dataset. According to human assessments, the system exhibited a 91\%
  accuracy rate in generating reasonable action descriptions and a 75\% accuracy
  rate in executing the correct actions for single-step instructions on iOS.
  Additionally, we evaluate the model on a subset of an Android screen navigation
  dataset, where the model outperforms previous GUI navigators in a zero-shot
  fashion. Our benchmark and detailed analyses aim to lay a robust groundwork for
  future research into the GUI navigation task. The project page is at
  https://github.com/zzxslp/MM-Navigator.
paper_authors._id: 6552e9bcb2d47c6675c91bbc
paper_authors.name: An Yan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-22T06:55:46.871Z
paper_authors._id: 634dfb9bbe5a827d48749f39
paper_authors.avatarUrl: /avatars/a8df0eeb1df39f3935dd686a94768f30.svg
paper_authors.fullname: An Yan
paper_authors.type: user
paper_authors.user: zzxslp
paper_authors._id: 6552e9bcb2d47c6675c91bbd
paper_authors.name: Zhengyuan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:34:14.297Z
paper_authors._id: 630713411801ecc7d2592a7c
paper_authors.avatarUrl: /avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
paper_authors.fullname: Zhengyuan Yang
paper_authors.type: user
paper_authors.user: zyang39
paper_authors._id: 6552e9bcb2d47c6675c91bbe
paper_authors.name: Wanrong Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-14T09:36:54.022Z
paper_authors._id: 633a7e91c8eb7b089034fb3c
paper_authors.avatarUrl: /avatars/0e03d80aa469de131cc17bb0c5c016b9.svg
paper_authors.fullname: Wanrong Zhu
paper_authors.type: user
paper_authors.user: VegB
paper_authors._id: 6552e9bcb2d47c6675c91bbf
paper_authors.name: Kevin Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-14T09:36:50.637Z
paper_authors._id: 6298fd95b58e71e2ac9f3ad8
paper_authors.avatarUrl: /avatars/7d34644d537bc5c17cf1e4ce4095355c.svg
paper_authors.fullname: Kevin Lin
paper_authors.type: user
paper_authors.user: kevinlin311tw
paper_authors._id: 6552e9bcb2d47c6675c91bc0
paper_authors.name: Linjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:34:30.187Z
paper_authors._id: 63db16fff03c3d71ef397206
paper_authors.avatarUrl: /avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
paper_authors.fullname: Linjie Li
paper_authors.type: user
paper_authors.user: linjieli222
paper_authors._id: 6552e9bcb2d47c6675c91bc1
paper_authors.name: Jianfeng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:35:00.082Z
paper_authors._id: 641404e74e5305c14f269b3c
paper_authors.avatarUrl: /avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
paper_authors.fullname: Jianfeng Wang
paper_authors.type: user
paper_authors.user: amsword8
paper_authors._id: 6552e9bcb2d47c6675c91bc2
paper_authors.name: Jianwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:35:18.221Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_authors._id: 6552e9bcb2d47c6675c91bc3
paper_authors.name: Yiwu Zhong
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-11-14T04:43:45.018Z
paper_authors._id: 62b4fe7ab25cb80fcf2ffd66
paper_authors.avatarUrl: /avatars/22f6a05cdbf4224d29ec9259c9fdd7a4.svg
paper_authors.fullname: Yiwu Zhong
paper_authors.type: user
paper_authors.user: YiwuZhong
paper_authors._id: 6552e9bcb2d47c6675c91bc4
paper_authors.name: Julian McAuley
paper_authors._id: 6552e9bcb2d47c6675c91bc5
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:35:32.890Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 6552e9bcb2d47c6675c91bc6
paper_authors.name: Zicheng Liu
paper_authors._id: 6552e9bcb2d47c6675c91bc7
paper_authors.name: Lijuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:35:48.322Z
paper_authors._id: 6413521d4e5305c14f22e110
paper_authors.avatarUrl: /avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
paper_authors.fullname: Lijuan Wang
paper_authors.type: user
paper_authors.user: Lijuan
paper_title: GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone
    GUI Navigation
paper_summary: We present MM-Navigator, a GPT-4V-based agent for the smartphone graphical
  user interface (GUI) navigation task. MM-Navigator can interact with a
  smartphone screen as human users, and determine subsequent actions to fulfill
  given instructions. Our findings demonstrate that large multimodal models
  (LMMs), specifically GPT-4V, excel in zero-shot GUI navigation through its
  advanced screen interpretation, action reasoning, and precise action
  localization capabilities. We first benchmark MM-Navigator on our collected iOS
  screen dataset. According to human assessments, the system exhibited a 91\%
  accuracy rate in generating reasonable action descriptions and a 75\% accuracy
  rate in executing the correct actions for single-step instructions on iOS.
  Additionally, we evaluate the model on a subset of an Android screen navigation
  dataset, where the model outperforms previous GUI navigators in a zero-shot
  fashion. Our benchmark and detailed analyses aim to lay a robust groundwork for
  future research into the GUI navigation task. The project page is at
  https://github.com/zzxslp/MM-Navigator.
paper_ai_keywords: GPT-4V GUI navigation screen interpretation action reasoning action localization large multimodal models (LMMs) iOS screen dataset Android screen navigation dataset zero-shot navigation