DigiRL: Training In-The-Wild Device-Control Agents with Autonomous
  Reinforcement Learning
Training corpuses for vision language models (VLMs) typically lack sufficient
amounts of decision-centric data. This renders off-the-shelf VLMs sub-optimal
for decision-making tasks such as in-the-wild device control through graphical
user interfaces (GUIs). While training with static demonstrations has shown
some promise, we show that such methods fall short for controlling real GUIs
due to their failure to deal with real-world stochasticity and non-stationarity
not captured in static observational data. This paper introduces a novel
autonomous RL approach, called DigiRL, for training in-the-wild device control
agents through fine-tuning a pre-trained VLM in two stages: offline RL to
initialize the model, followed by offline-to-online RL. To do this, we build a
scalable and parallelizable Android learning environment equipped with a
VLM-based evaluator and develop a simple yet effective RL approach for learning
in this domain. Our approach runs advantage-weighted RL with advantage
estimators enhanced to account for stochasticity along with an automatic
curriculum for deriving maximal learning signal. We demonstrate the
effectiveness of DigiRL using the Android-in-the-Wild (AitW) dataset, where our
1.3B VLM trained with RL achieves a 49.5% absolute improvement -- from 17.7 to
67.2% success rate -- over supervised fine-tuning with static human
demonstration data. These results significantly surpass not only the prior best
agents, including AppAgent with GPT-4V (8.3% success rate) and the 17B CogAgent
trained with AitW data (38.5%), but also the prior best autonomous RL approach
based on filtered behavior cloning (57.8%), thereby establishing a new
state-of-the-art for digital agents for in-the-wild device control.
66745cfe299133baa5b817a8
Hao Bai
claimed_verified
2025-02-25T14:22:50.617Z
62927c2e56fedc76e396b3ca
https://cdn-avatars.huggingface.co/v1/production/uploads/1678105603200-62927c2e56fedc76e396b3ca.jpeg
HAO BAI
user
JackBAI
66745cfe299133baa5b817a9
Yifei Zhou
claimed_verified
2024-12-18T12:38:44.412Z
64fd0229e0dc35986bd3c0e5
/avatars/94f5698f9104dad7288edb4460026fd8.svg
Yifei Zhou
user
yifeizhou
66745cfe299133baa5b817aa
Mert Cemri
66745cfe299133baa5b817ab
Jiayi Pan
admin_assigned
2024-06-21T11:43:25.497Z
61568f37272f2d87a99ba884
https://cdn-avatars.huggingface.co/v1/production/uploads/61568f37272f2d87a99ba884/lgvkl5f0rEyiQRVU5FE32.png
Jiayi Pan
user
Jiayi-Pan
66745cfe299133baa5b817ac
Alane Suhr
claimed_verified
2025-01-02T10:19:46.809Z
6611e6e1188ff298b0dd0b79
/avatars/3a495283955ec9e06e1829c7eb2cd9a4.svg
Alane Suhr
user
alsuhr
66745cfe299133baa5b817ad
Sergey Levine
admin_assigned
2024-06-21T11:47:52.402Z
665ce54120a307a3754849dd
/avatars/e698726e9be61dd50ce2efe372ed5dac.svg
Sergey Levine
user
svlevine
66745cfe299133baa5b817ae
Aviral Kumar
admin_assigned
2024-06-21T11:48:05.505Z
64879130cc0e54f272b5dbcc
/avatars/ead1211a56a4e5c14cb8f6b2ea59c0e6.svg
Aviral Kumar
user
aviralkumar
DigiRL: Training In-The-Wild Device-Control Agents with Autonomous
  Reinforcement Learning
Training corpuses for vision language models (VLMs) typically lack sufficient
amounts of decision-centric data. This renders off-the-shelf VLMs sub-optimal
for decision-making tasks such as in-the-wild device control through graphical
user interfaces (GUIs). While training with static demonstrations has shown
some promise, we show that such methods fall short for controlling real GUIs
due to their failure to deal with real-world stochasticity and non-stationarity
not captured in static observational data. This paper introduces a novel
autonomous RL approach, called DigiRL, for training in-the-wild device control
agents through fine-tuning a pre-trained VLM in two stages: offline RL to
initialize the model, followed by offline-to-online RL. To do this, we build a
scalable and parallelizable Android learning environment equipped with a
VLM-based evaluator and develop a simple yet effective RL approach for learning
in this domain. Our approach runs advantage-weighted RL with advantage
estimators enhanced to account for stochasticity along with an automatic
curriculum for deriving maximal learning signal. We demonstrate the
effectiveness of DigiRL using the Android-in-the-Wild (AitW) dataset, where our
1.3B VLM trained with RL achieves a 49.5% absolute improvement -- from 17.7 to
67.2% success rate -- over supervised fine-tuning with static human
demonstration data. These results significantly surpass not only the prior best
agents, including AppAgent with GPT-4V (8.3% success rate) and the 17B CogAgent
trained with AitW data (38.5%), but also the prior best autonomous RL approach
based on filtered behavior cloning (57.8%), thereby establishing a new
state-of-the-art for digital agents for in-the-wild device control.
A novel autonomous RL approach, DigiRL, fine-tunes a pre-trained VLM for in-the-wild device control, achieving superior performance over existing methods.
autonomous RL decision-making tasks graphical user interfaces GUIs static demonstrations real-world stochasticity non-stationarity offline RL offline-to-online RL VLM-based evaluator advantage-weighted RL advantage estimators automatic curriculum Android-in-the-Wild dataset VLM success rate AppAgent CogAgent filtered behavior cloning