Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual
  Perception
Mobile device agent based on Multimodal Large Language Models (MLLM) is
becoming a popular application. In this paper, we introduce Mobile-Agent, an
autonomous multi-modal mobile device agent. Mobile-Agent first leverages visual
perception tools to accurately identify and locate both the visual and textual
elements within the app's front-end interface. Based on the perceived vision
context, it then autonomously plans and decomposes the complex operation task,
and navigates the mobile Apps through operations step by step. Different from
previous solutions that rely on XML files of Apps or mobile system metadata,
Mobile-Agent allows for greater adaptability across diverse mobile operating
environments in a vision-centric way, thereby eliminating the necessity for
system-specific customizations. To assess the performance of Mobile-Agent, we
introduced Mobile-Eval, a benchmark for evaluating mobile device operations.
Based on Mobile-Eval, we conducted a comprehensive evaluation of Mobile-Agent.
The experimental results indicate that Mobile-Agent achieved remarkable
accuracy and completion rates. Even with challenging instructions, such as
multi-app operations, Mobile-Agent can still complete the requirements. Code
and model will be open-sourced at https://github.com/X-PLUG/MobileAgent.
65b86e5d45d7cf731f836a12
Junyang Wang
admin_assigned
2024-01-30T08:19:05.778Z
6438f6415aa69077ffb16942
/avatars/c83dbd3e10e88db97c2a86092bad5917.svg
Junyang Wang
user
junyangwang0410
65b86e5d45d7cf731f836a13
Haiyang Xu
claimed_verified
2024-01-30T08:06:34.075Z
645b10e80c73ea27d13f7aca
/avatars/95e565306472a15067440b5b43e07a6f.svg
xuhaiyang
user
xhyandwyy
65b86e5d45d7cf731f836a14
Jiabo Ye
65b86e5d45d7cf731f836a15
Ming Yan
claimed_verified
2024-01-30T08:06:19.833Z
64771cfdd7cf39f2e9381aa9
/avatars/48adf00c3b653df02628f80511639e19.svg
Ming
user
MingYan123
65b86e5d45d7cf731f836a16
Weizhou Shen
admin_assigned
2024-01-30T08:19:28.485Z
64777a346e6c7ac608c1e9bf
/avatars/b0e65ba781c90c2560606eb5467101eb.svg
Weizhou Shen
user
shenwzh3
65b86e5d45d7cf731f836a17
Ji Zhang
65b86e5d45d7cf731f836a18
Fei Huang
admin_assigned
2024-01-30T08:20:49.620Z
635b8b6a37c6a2c12e2cce00
/avatars/229fb72180529141515d1df797b33709.svg
Fei Huang
user
hzhwcmhf
65b86e5d45d7cf731f836a19
Jitao Sang
claimed_verified
2024-12-04T09:47:58.287Z
6494457c6339264dd78bcb95
/avatars/d87842251f1a43f50cc827f0e2a995ee.svg
sdzy
user
sdzy
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual
  Perception
Mobile device agent based on Multimodal Large Language Models (MLLM) is
becoming a popular application. In this paper, we introduce Mobile-Agent, an
autonomous multi-modal mobile device agent. Mobile-Agent first leverages visual
perception tools to accurately identify and locate both the visual and textual
elements within the app's front-end interface. Based on the perceived vision
context, it then autonomously plans and decomposes the complex operation task,
and navigates the mobile Apps through operations step by step. Different from
previous solutions that rely on XML files of Apps or mobile system metadata,
Mobile-Agent allows for greater adaptability across diverse mobile operating
environments in a vision-centric way, thereby eliminating the necessity for
system-specific customizations. To assess the performance of Mobile-Agent, we
introduced Mobile-Eval, a benchmark for evaluating mobile device operations.
Based on Mobile-Eval, we conducted a comprehensive evaluation of Mobile-Agent.
The experimental results indicate that Mobile-Agent achieved remarkable
accuracy and completion rates. Even with challenging instructions, such as
multi-app operations, Mobile-Agent can still complete the requirements. Code
and model will be open-sourced at https://github.com/X-PLUG/MobileAgent.
Mobile-Agent, a vision-centric multimodal large language model agent for mobile devices, autonomously performs complex tasks without relying on system-specific customizations and achieves high accuracy and adaptability.
Multimodal Large Language Models MLLM mobile device agent visual perception tools autonomous planning vision-centric Mobile-Eval benchmark mobile device operations