title: VEM: Environment-Free Exploration for Training GUI Agent with Value
    Environment Model
summary: Training Vision-Language Models (VLMs) for Graphical User Interfaces (GUI)
  agents via Reinforcement Learning (RL) faces critical challenges:
  environment-based RL requires costly interactions, while environment-free
  methods struggle with distribution shift and reward generalization. We propose
  an environment-free RL framework that decouples value estimation from policy
  optimization by leveraging a pretrained Value Environment Model (VEM). VEM
  predicts state-action values directly from offline data, distilling human-like
  priors about GUI interaction outcomes without requiring next-state prediction
  or environmental feedback. This avoids compounding errors and enhances
  resilience to UI changes by focusing on semantic reasoning (e.g., Does this
  action advance the user's goal?). The framework operates in two stages: (1)
  pretraining VEM to estimate long-term action utilities and (2) guiding policy
  exploration with frozen VEM signals, enabling layout-agnostic GUI automation.
  Evaluated on Android-in-the-Wild benchmarks, VEM achieves state-of-the-art
  performance in both offline and online settings, outperforming environment-free
  baselines significantly and matching environment-based approaches without
  interaction costs. Importantly, VEM demonstrates that semantic-aware value
  estimation can achieve comparable performance with online-trained methods.
paper_authors._id: 67bfd5d2381f8fcb67e5ad36
paper_authors.name: Jiani Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-02T20:18:18.436Z
paper_authors._id: 64531f631a57e1179c203e6b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64531f631a57e1179c203e6b/C_J7pXFLqoJoHYPPhK3J9.jpeg
paper_authors.fullname: zjn
paper_authors.type: user
paper_authors.user: garlicisnotmyfavor
paper_authors._id: 67bfd5d2381f8fcb67e5ad37
paper_authors.name: Lu Wang
paper_authors._id: 67bfd5d2381f8fcb67e5ad38
paper_authors.name: Fangkai Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-27T09:14:57.452Z
paper_authors._id: 669dcf6200970c3b27aafa5d
paper_authors.avatarUrl: /avatars/bb9ed5ff86326fdaeb184c6b0e40f74f.svg
paper_authors.fullname: kaikai yang
paper_authors.type: user
paper_authors.user: keanudicap
paper_authors._id: 67bfd5d2381f8fcb67e5ad39
paper_authors.name: Chaoyun Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-27T09:14:59.653Z
paper_authors._id: 654dbac9938fbf1e696be8aa
paper_authors.avatarUrl: /avatars/b3c4035c48169c1bfb04a439fce3499f.svg
paper_authors.fullname: Chaoyun Zhang
paper_authors.type: user
paper_authors.user: vyokky
paper_authors._id: 67bfd5d2381f8fcb67e5ad3a
paper_authors.name: Lingrui Mei
paper_authors._id: 67bfd5d2381f8fcb67e5ad3b
paper_authors.name: Wenjie Yin
paper_authors._id: 67bfd5d2381f8fcb67e5ad3c
paper_authors.name: Qingwei Lin
paper_authors._id: 67bfd5d2381f8fcb67e5ad3d
paper_authors.name: Dongmei Zhang
paper_authors._id: 67bfd5d2381f8fcb67e5ad3e
paper_authors.name: Saravan Rajmohan
paper_authors._id: 67bfd5d2381f8fcb67e5ad3f
paper_authors.name: Qi Zhang
paper_title: VEM: Environment-Free Exploration for Training GUI Agent with Value
    Environment Model
paper_summary: Training Vision-Language Models (VLMs) for Graphical User Interfaces (GUI)
  agents via Reinforcement Learning (RL) faces critical challenges:
  environment-based RL requires costly interactions, while environment-free
  methods struggle with distribution shift and reward generalization. We propose
  an environment-free RL framework that decouples value estimation from policy
  optimization by leveraging a pretrained Value Environment Model (VEM). VEM
  predicts state-action values directly from offline data, distilling human-like
  priors about GUI interaction outcomes without requiring next-state prediction
  or environmental feedback. This avoids compounding errors and enhances
  resilience to UI changes by focusing on semantic reasoning (e.g., Does this
  action advance the user's goal?). The framework operates in two stages: (1)
  pretraining VEM to estimate long-term action utilities and (2) guiding policy
  exploration with frozen VEM signals, enabling layout-agnostic GUI automation.
  Evaluated on Android-in-the-Wild benchmarks, VEM achieves state-of-the-art
  performance in both offline and online settings, outperforming environment-free
  baselines significantly and matching environment-based approaches without
  interaction costs. Importantly, VEM demonstrates that semantic-aware value
  estimation can achieve comparable performance with online-trained methods.
paper_ai_summary: The proposed environment-free RL framework uses a pretrained Value Environment Model (VEM) for offline value estimation and policy optimization, achieving high performance in GUI automation with minimal interaction costs.
paper_ai_keywords: Reinforcement Learning Value Environment Model VEM semantic reasoning offline data GUI automation Android-in-the-Wild benchmarks