title: Lightweight Neural App Control
summary: This paper introduces a novel mobile phone control architecture, termed ``app
  agents", for efficient interactions and controls across various Android apps.
  The proposed Lightweight Multi-modal App Control (LiMAC) takes as input a
  textual goal and a sequence of past mobile observations, such as screenshots
  and corresponding UI trees, to generate precise actions. To address the
  computational constraints inherent to smartphones, within LiMAC, we introduce a
  small Action Transformer (AcT) integrated with a fine-tuned vision-language
  model (VLM) for real-time decision-making and task execution. We evaluate LiMAC
  on two open-source mobile control datasets, demonstrating the superior
  performance of our small-form-factor approach against fine-tuned versions of
  open-source VLMs, such as Florence2 and Qwen2-VL. It also significantly
  outperforms prompt engineering baselines utilising closed-source foundation
  models like GPT-4o. More specifically, LiMAC increases the overall action
  accuracy by up to 19% compared to fine-tuned VLMs, and up to 42% compared to
  prompt-engineering baselines.
paper_authors._id: 6719e4bd2c519a08bd42b9d5
paper_authors.name: Filippos Christianos
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:34:29.693Z
paper_authors._id: 64f46b681d337935d0495d4d
paper_authors.avatarUrl: /avatars/cce5a4910617931fb13062b832e14ef8.svg
paper_authors.fullname: Filippos Christianos
paper_authors.type: user
paper_authors.user: semitable
paper_authors._id: 6719e4bd2c519a08bd42b9d6
paper_authors.name: Georgios Papoudakis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:34:34.954Z
paper_authors._id: 66e451f4c3b3a128bd078bdf
paper_authors.avatarUrl: /avatars/624e2520129f408874f227df04025ec7.svg
paper_authors.fullname: Georgios Papoudakis
paper_authors.type: user
paper_authors.user: gpap
paper_authors._id: 6719e4bd2c519a08bd42b9d7
paper_authors.name: Thomas Coste
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:34:41.324Z
paper_authors._id: 64ca5357ead94891d1d53cd1
paper_authors.avatarUrl: /avatars/187073352d7767f18d5fd50a6ca69fb5.svg
paper_authors.fullname: Thomas COSTE
paper_authors.type: user
paper_authors.user: Fahren24
paper_authors._id: 6719e4bd2c519a08bd42b9d8
paper_authors.name: Jianye Hao
paper_authors._id: 6719e4bd2c519a08bd42b9d9
paper_authors.name: Jun Wang
paper_authors._id: 6719e4bd2c519a08bd42b9da
paper_authors.name: Kun Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-24T08:34:58.709Z
paper_authors._id: 66840c818f6c78ebd41f86ca
paper_authors.avatarUrl: /avatars/626ffa3d262c02f964f15fa420af414c.svg
paper_authors.fullname: Kun Shao
paper_authors.type: user
paper_authors.user: ShaoKun-HW
paper_title: Lightweight Neural App Control
paper_summary: This paper introduces a novel mobile phone control architecture, termed ``app
  agents", for efficient interactions and controls across various Android apps.
  The proposed Lightweight Multi-modal App Control (LiMAC) takes as input a
  textual goal and a sequence of past mobile observations, such as screenshots
  and corresponding UI trees, to generate precise actions. To address the
  computational constraints inherent to smartphones, within LiMAC, we introduce a
  small Action Transformer (AcT) integrated with a fine-tuned vision-language
  model (VLM) for real-time decision-making and task execution. We evaluate LiMAC
  on two open-source mobile control datasets, demonstrating the superior
  performance of our small-form-factor approach against fine-tuned versions of
  open-source VLMs, such as Florence2 and Qwen2-VL. It also significantly
  outperforms prompt engineering baselines utilising closed-source foundation
  models like GPT-4o. More specifically, LiMAC increases the overall action
  accuracy by up to 19% compared to fine-tuned VLMs, and up to 42% compared to
  prompt-engineering baselines.
paper_ai_keywords: app agents Lightweight Multi-modal App Control (LiMAC) Action Transformer (AcT) fine-tuned vision-language model (VLM) real-time decision-making task execution mobile control datasets Florence2 Qwen2-VL GPT-4o action accuracy