title: DigiRL: Training In-The-Wild Device-Control Agents with Autonomous
    Reinforcement Learning
summary: Training corpuses for vision language models (VLMs) typically lack sufficient
  amounts of decision-centric data. This renders off-the-shelf VLMs sub-optimal
  for decision-making tasks such as in-the-wild device control through graphical
  user interfaces (GUIs). While training with static demonstrations has shown
  some promise, we show that such methods fall short for controlling real GUIs
  due to their failure to deal with real-world stochasticity and non-stationarity
  not captured in static observational data. This paper introduces a novel
  autonomous RL approach, called DigiRL, for training in-the-wild device control
  agents through fine-tuning a pre-trained VLM in two stages: offline RL to
  initialize the model, followed by offline-to-online RL. To do this, we build a
  scalable and parallelizable Android learning environment equipped with a
  VLM-based evaluator and develop a simple yet effective RL approach for learning
  in this domain. Our approach runs advantage-weighted RL with advantage
  estimators enhanced to account for stochasticity along with an automatic
  curriculum for deriving maximal learning signal. We demonstrate the
  effectiveness of DigiRL using the Android-in-the-Wild (AitW) dataset, where our
  1.3B VLM trained with RL achieves a 49.5% absolute improvement -- from 17.7 to
  67.2% success rate -- over supervised fine-tuning with static human
  demonstration data. These results significantly surpass not only the prior best
  agents, including AppAgent with GPT-4V (8.3% success rate) and the 17B CogAgent
  trained with AitW data (38.5%), but also the prior best autonomous RL approach
  based on filtered behavior cloning (57.8%), thereby establishing a new
  state-of-the-art for digital agents for in-the-wild device control.
paper_authors._id: 66745cfe299133baa5b817a8
paper_authors.name: Hao Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-25T14:22:50.617Z
paper_authors._id: 62927c2e56fedc76e396b3ca
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678105603200-62927c2e56fedc76e396b3ca.jpeg
paper_authors.fullname: HAO BAI
paper_authors.type: user
paper_authors.user: JackBAI
paper_authors._id: 66745cfe299133baa5b817a9
paper_authors.name: Yifei Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-18T12:38:44.412Z
paper_authors._id: 64fd0229e0dc35986bd3c0e5
paper_authors.avatarUrl: /avatars/94f5698f9104dad7288edb4460026fd8.svg
paper_authors.fullname: Yifei Zhou
paper_authors.type: user
paper_authors.user: yifeizhou
paper_authors._id: 66745cfe299133baa5b817aa
paper_authors.name: Mert Cemri
paper_authors._id: 66745cfe299133baa5b817ab
paper_authors.name: Jiayi Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T11:43:25.497Z
paper_authors._id: 61568f37272f2d87a99ba884
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61568f37272f2d87a99ba884/lgvkl5f0rEyiQRVU5FE32.png
paper_authors.fullname: Jiayi Pan
paper_authors.type: user
paper_authors.user: Jiayi-Pan
paper_authors._id: 66745cfe299133baa5b817ac
paper_authors.name: Alane Suhr
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-02T10:19:46.809Z
paper_authors._id: 6611e6e1188ff298b0dd0b79
paper_authors.avatarUrl: /avatars/3a495283955ec9e06e1829c7eb2cd9a4.svg
paper_authors.fullname: Alane Suhr
paper_authors.type: user
paper_authors.user: alsuhr
paper_authors._id: 66745cfe299133baa5b817ad
paper_authors.name: Sergey Levine
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T11:47:52.402Z
paper_authors._id: 665ce54120a307a3754849dd
paper_authors.avatarUrl: /avatars/e698726e9be61dd50ce2efe372ed5dac.svg
paper_authors.fullname: Sergey Levine
paper_authors.type: user
paper_authors.user: svlevine
paper_authors._id: 66745cfe299133baa5b817ae
paper_authors.name: Aviral Kumar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T11:48:05.505Z
paper_authors._id: 64879130cc0e54f272b5dbcc
paper_authors.avatarUrl: /avatars/ead1211a56a4e5c14cb8f6b2ea59c0e6.svg
paper_authors.fullname: Aviral Kumar
paper_authors.type: user
paper_authors.user: aviralkumar
paper_title: DigiRL: Training In-The-Wild Device-Control Agents with Autonomous
    Reinforcement Learning
paper_summary: Training corpuses for vision language models (VLMs) typically lack sufficient
  amounts of decision-centric data. This renders off-the-shelf VLMs sub-optimal
  for decision-making tasks such as in-the-wild device control through graphical
  user interfaces (GUIs). While training with static demonstrations has shown
  some promise, we show that such methods fall short for controlling real GUIs
  due to their failure to deal with real-world stochasticity and non-stationarity
  not captured in static observational data. This paper introduces a novel
  autonomous RL approach, called DigiRL, for training in-the-wild device control
  agents through fine-tuning a pre-trained VLM in two stages: offline RL to
  initialize the model, followed by offline-to-online RL. To do this, we build a
  scalable and parallelizable Android learning environment equipped with a
  VLM-based evaluator and develop a simple yet effective RL approach for learning
  in this domain. Our approach runs advantage-weighted RL with advantage
  estimators enhanced to account for stochasticity along with an automatic
  curriculum for deriving maximal learning signal. We demonstrate the
  effectiveness of DigiRL using the Android-in-the-Wild (AitW) dataset, where our
  1.3B VLM trained with RL achieves a 49.5% absolute improvement -- from 17.7 to
  67.2% success rate -- over supervised fine-tuning with static human
  demonstration data. These results significantly surpass not only the prior best
  agents, including AppAgent with GPT-4V (8.3% success rate) and the 17B CogAgent
  trained with AitW data (38.5%), but also the prior best autonomous RL approach
  based on filtered behavior cloning (57.8%), thereby establishing a new
  state-of-the-art for digital agents for in-the-wild device control.
paper_ai_keywords: autonomous RL DigiRL offline RL advantage-weighted RL advantage estimators automatic curriculum Android learning environment VLM-based evaluator Android-in-the-Wild (AitW) dataset filtered behavior cloning AppAgent GPT-4V CogAgent