title: LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration
    Benchmark
summary: Mobile GUI agents show promise in automating tasks but face generalization
  challenges in diverse real-world scenarios. Traditional approaches using
  pre-training or fine-tuning with massive datasets struggle with the diversity
  of mobile applications and user-specific tasks. We propose enhancing mobile GUI
  agent capabilities through human demonstrations, focusing on improving
  performance in unseen scenarios rather than pursuing universal generalization
  through larger datasets. To realize this paradigm, we introduce LearnGUI, the
  first comprehensive dataset specifically designed for studying
  demonstration-based learning in mobile GUI agents, comprising 2,252 offline
  tasks and 101 online tasks with high-quality human demonstrations. We further
  develop LearnAct, a sophisticated multi-agent framework that automatically
  extracts knowledge from demonstrations to enhance task completion. This
  framework integrates three specialized agents: DemoParser for knowledge
  extraction, KnowSeeker for relevant knowledge retrieval, and ActExecutor for
  demonstration-enhanced task execution. Our experimental results show
  significant performance gains in both offline and online evaluations. In
  offline assessments, a single demonstration improves model performance,
  increasing Gemini-1.5-Pro's accuracy from 19.3% to 51.7%. In online
  evaluations, our framework enhances UI-TARS-7B-SFT's task success rate from
  18.1% to 32.8%. LearnAct framework and LearnGUI benchmark establish
  demonstration-based learning as a promising direction for more adaptable,
  personalized, and deployable mobile GUI agents.
paper_authors._id: 680662b97415e191e3579b9e
paper_authors.name: Guangyi Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T09:52:04.194Z
paper_authors._id: 64d761b98ebc40443831f82a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d761b98ebc40443831f82a/DHBOtOstiFp2-lDY6b9gb.png
paper_authors.fullname: Guangyi Liu
paper_authors.type: user
paper_authors.user: lgy0404
paper_authors._id: 680662b97415e191e3579b9f
paper_authors.name: Pengxiang Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T09:52:01.390Z
paper_authors._id: 65a088f4300957620ba45c70
paper_authors.avatarUrl: /avatars/56ed45e10d3455531979f30881b2d3f9.svg
paper_authors.fullname: pengxiang zhao
paper_authors.type: user
paper_authors.user: Pengxiangzhao
paper_authors._id: 680662b97415e191e3579ba0
paper_authors.name: Liang Liu
paper_authors._id: 680662b97415e191e3579ba1
paper_authors.name: Zhiming Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T11:01:08.419Z
paper_authors._id: 67c26500918bb8d89b01d6a3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/OJROx6Q4Z2rnW5eOiqvm3.png
paper_authors.fullname: Zhiming Chen
paper_authors.type: user
paper_authors.user: zhimingc
paper_authors._id: 680662b97415e191e3579ba2
paper_authors.name: Yuxiang Chai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T11:01:01.122Z
paper_authors._id: 6458ce236fa580137af5aa95
paper_authors.avatarUrl: /avatars/db65a7332e375eb5daad5c1b076b1e3b.svg
paper_authors.fullname: Yuxiang Chai
paper_authors.type: user
paper_authors.user: Yuxiang007
paper_authors._id: 680662b97415e191e3579ba3
paper_authors.name: Shuai Ren
paper_authors._id: 680662b97415e191e3579ba4
paper_authors.name: Hao Wang
paper_authors._id: 680662b97415e191e3579ba5
paper_authors.name: Shibo He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T11:00:42.823Z
paper_authors._id: 66604211253289136b6c42d6
paper_authors.avatarUrl: /avatars/b2ea4870d45efea01d4c2f34a2cd96bd.svg
paper_authors.fullname: SHIBO HE
paper_authors.type: user
paper_authors.user: dkeeeee
paper_authors._id: 680662b97415e191e3579ba6
paper_authors.name: Wenchao Meng
paper_title: LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration
    Benchmark
paper_summary: Mobile GUI agents show promise in automating tasks but face generalization
  challenges in diverse real-world scenarios. Traditional approaches using
  pre-training or fine-tuning with massive datasets struggle with the diversity
  of mobile applications and user-specific tasks. We propose enhancing mobile GUI
  agent capabilities through human demonstrations, focusing on improving
  performance in unseen scenarios rather than pursuing universal generalization
  through larger datasets. To realize this paradigm, we introduce LearnGUI, the
  first comprehensive dataset specifically designed for studying
  demonstration-based learning in mobile GUI agents, comprising 2,252 offline
  tasks and 101 online tasks with high-quality human demonstrations. We further
  develop LearnAct, a sophisticated multi-agent framework that automatically
  extracts knowledge from demonstrations to enhance task completion. This
  framework integrates three specialized agents: DemoParser for knowledge
  extraction, KnowSeeker for relevant knowledge retrieval, and ActExecutor for
  demonstration-enhanced task execution. Our experimental results show
  significant performance gains in both offline and online evaluations. In
  offline assessments, a single demonstration improves model performance,
  increasing Gemini-1.5-Pro's accuracy from 19.3% to 51.7%. In online
  evaluations, our framework enhances UI-TARS-7B-SFT's task success rate from
  18.1% to 32.8%. LearnAct framework and LearnGUI benchmark establish
  demonstration-based learning as a promising direction for more adaptable,
  personalized, and deployable mobile GUI agents.
paper_ai_summary: The LearnAct framework and LearnGUI dataset improve mobile GUI agent performance through demonstration-based learning, enhancing task success rates in both offline and online evaluations.
paper_ai_keywords: LearnGUI LearnAct multi-agent framework DemoParser KnowSeeker ActExecutor demonstration-based learning