title: Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual
    Perception
summary: Mobile device agent based on Multimodal Large Language Models (MLLM) is
  becoming a popular application. In this paper, we introduce Mobile-Agent, an
  autonomous multi-modal mobile device agent. Mobile-Agent first leverages visual
  perception tools to accurately identify and locate both the visual and textual
  elements within the app's front-end interface. Based on the perceived vision
  context, it then autonomously plans and decomposes the complex operation task,
  and navigates the mobile Apps through operations step by step. Different from
  previous solutions that rely on XML files of Apps or mobile system metadata,
  Mobile-Agent allows for greater adaptability across diverse mobile operating
  environments in a vision-centric way, thereby eliminating the necessity for
  system-specific customizations. To assess the performance of Mobile-Agent, we
  introduced Mobile-Eval, a benchmark for evaluating mobile device operations.
  Based on Mobile-Eval, we conducted a comprehensive evaluation of Mobile-Agent.
  The experimental results indicate that Mobile-Agent achieved remarkable
  accuracy and completion rates. Even with challenging instructions, such as
  multi-app operations, Mobile-Agent can still complete the requirements. Code
  and model will be open-sourced at https://github.com/X-PLUG/MobileAgent.
paper_authors._id: 65b86e5d45d7cf731f836a12
paper_authors.name: Junyang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-30T08:19:05.778Z
paper_authors._id: 6438f6415aa69077ffb16942
paper_authors.avatarUrl: /avatars/c83dbd3e10e88db97c2a86092bad5917.svg
paper_authors.fullname: Junyang Wang
paper_authors.type: user
paper_authors.user: junyangwang0410
paper_authors._id: 65b86e5d45d7cf731f836a13
paper_authors.name: Haiyang Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-30T08:06:34.075Z
paper_authors._id: 645b10e80c73ea27d13f7aca
paper_authors.avatarUrl: /avatars/95e565306472a15067440b5b43e07a6f.svg
paper_authors.fullname: xuhaiyang
paper_authors.type: user
paper_authors.user: xhyandwyy
paper_authors._id: 65b86e5d45d7cf731f836a14
paper_authors.name: Jiabo Ye
paper_authors._id: 65b86e5d45d7cf731f836a15
paper_authors.name: Ming Yan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-30T08:06:19.833Z
paper_authors._id: 64771cfdd7cf39f2e9381aa9
paper_authors.avatarUrl: /avatars/48adf00c3b653df02628f80511639e19.svg
paper_authors.fullname: Ming
paper_authors.type: user
paper_authors.user: MingYan123
paper_authors._id: 65b86e5d45d7cf731f836a16
paper_authors.name: Weizhou Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-30T08:19:28.485Z
paper_authors._id: 64777a346e6c7ac608c1e9bf
paper_authors.avatarUrl: /avatars/b0e65ba781c90c2560606eb5467101eb.svg
paper_authors.fullname: Weizhou Shen
paper_authors.type: user
paper_authors.user: shenwzh3
paper_authors._id: 65b86e5d45d7cf731f836a17
paper_authors.name: Ji Zhang
paper_authors._id: 65b86e5d45d7cf731f836a18
paper_authors.name: Fei Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-30T08:20:49.620Z
paper_authors._id: 635b8b6a37c6a2c12e2cce00
paper_authors.avatarUrl: /avatars/229fb72180529141515d1df797b33709.svg
paper_authors.fullname: Fei Huang
paper_authors.type: user
paper_authors.user: hzhwcmhf
paper_authors._id: 65b86e5d45d7cf731f836a19
paper_authors.name: Jitao Sang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:47:58.287Z
paper_authors._id: 6494457c6339264dd78bcb95
paper_authors.avatarUrl: /avatars/d87842251f1a43f50cc827f0e2a995ee.svg
paper_authors.fullname: sdzy
paper_authors.type: user
paper_authors.user: sdzy
paper_title: Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual
    Perception
paper_summary: Mobile device agent based on Multimodal Large Language Models (MLLM) is
  becoming a popular application. In this paper, we introduce Mobile-Agent, an
  autonomous multi-modal mobile device agent. Mobile-Agent first leverages visual
  perception tools to accurately identify and locate both the visual and textual
  elements within the app's front-end interface. Based on the perceived vision
  context, it then autonomously plans and decomposes the complex operation task,
  and navigates the mobile Apps through operations step by step. Different from
  previous solutions that rely on XML files of Apps or mobile system metadata,
  Mobile-Agent allows for greater adaptability across diverse mobile operating
  environments in a vision-centric way, thereby eliminating the necessity for
  system-specific customizations. To assess the performance of Mobile-Agent, we
  introduced Mobile-Eval, a benchmark for evaluating mobile device operations.
  Based on Mobile-Eval, we conducted a comprehensive evaluation of Mobile-Agent.
  The experimental results indicate that Mobile-Agent achieved remarkable
  accuracy and completion rates. Even with challenging instructions, such as
  multi-app operations, Mobile-Agent can still complete the requirements. Code
  and model will be open-sourced at https://github.com/X-PLUG/MobileAgent.
paper_ai_summary: Mobile-Agent, a vision-centric multimodal large language model agent for mobile devices, autonomously performs complex tasks without relying on system-specific customizations and achieves high accuracy and adaptability.
paper_ai_keywords: Multimodal Large Language Models MLLM mobile device agent visual perception tools autonomous planning vision-centric Mobile-Eval benchmark mobile device operations