Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error
  Diagnosis in GUI Automation
In recent years, Multimodal Large Language Models (MLLMs) have been
extensively utilized for multimodal reasoning tasks, including Graphical User
Interface (GUI) automation. Unlike general offline multimodal tasks, GUI
automation is executed in online interactive environments, necessitating
step-by-step decision-making based on real-time status of the environment. This
task has a lower tolerance for decision-making errors at each step, as any
mistakes may cumulatively disrupt the process and potentially lead to
irreversible outcomes like deletions or payments. To address these issues, we
introduce a pre-operative critic mechanism that provides effective feedback
prior to the actual execution, by reasoning about the potential outcome and
correctness of actions. Specifically, we propose a Suggestion-aware Gradient
Relative Policy Optimization (S-GRPO) strategy to construct our pre-operative
critic model GUI-Critic-R1, incorporating a novel suggestion reward to enhance
the reliability of the model's feedback. Furthermore, we develop a
reasoning-bootstrapping based data collection pipeline to create a
GUI-Critic-Train and a GUI-Critic-Test, filling existing gaps in GUI critic
data. Static experiments on the GUI-Critic-Test across both mobile and web
domains reveal that our GUI-Critic-R1 offers significant advantages in critic
accuracy compared to current MLLMs. Dynamic evaluation on GUI automation
benchmark further highlights the effectiveness and superiority of our model, as
evidenced by improved success rates and operational efficiency.
684921e342e4f9106973f3e7
Yuyang Wanyan
684921e342e4f9106973f3e8
Xi Zhang
684921e342e4f9106973f3e9
Haiyang Xu
claimed_verified
2025-06-11T16:02:16.976Z
645b10e80c73ea27d13f7aca
/avatars/95e565306472a15067440b5b43e07a6f.svg
xuhaiyang
user
xhyandwyy
684921e342e4f9106973f3ea
Haowei Liu
684921e342e4f9106973f3eb
Junyang Wang
684921e342e4f9106973f3ec
Jiabo Ye
684921e342e4f9106973f3ed
Yutong Kou
684921e342e4f9106973f3ee
Ming Yan
684921e342e4f9106973f3ef
Fei Huang
684921e342e4f9106973f3f0
Xiaoshan Yang
684921e342e4f9106973f3f1
Weiming Dong
684921e342e4f9106973f3f2
Changsheng Xu
Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error
  Diagnosis in GUI Automation
In recent years, Multimodal Large Language Models (MLLMs) have been
extensively utilized for multimodal reasoning tasks, including Graphical User
Interface (GUI) automation. Unlike general offline multimodal tasks, GUI
automation is executed in online interactive environments, necessitating
step-by-step decision-making based on real-time status of the environment. This
task has a lower tolerance for decision-making errors at each step, as any
mistakes may cumulatively disrupt the process and potentially lead to
irreversible outcomes like deletions or payments. To address these issues, we
introduce a pre-operative critic mechanism that provides effective feedback
prior to the actual execution, by reasoning about the potential outcome and
correctness of actions. Specifically, we propose a Suggestion-aware Gradient
Relative Policy Optimization (S-GRPO) strategy to construct our pre-operative
critic model GUI-Critic-R1, incorporating a novel suggestion reward to enhance
the reliability of the model's feedback. Furthermore, we develop a
reasoning-bootstrapping based data collection pipeline to create a
GUI-Critic-Train and a GUI-Critic-Test, filling existing gaps in GUI critic
data. Static experiments on the GUI-Critic-Test across both mobile and web
domains reveal that our GUI-Critic-R1 offers significant advantages in critic
accuracy compared to current MLLMs. Dynamic evaluation on GUI automation
benchmark further highlights the effectiveness and superiority of our model, as
evidenced by improved success rates and operational efficiency.
A pre-operative critic mechanism with Suggestion-aware Gradient Relative Policy Optimization enhances the reliability of multimodal reasoning tasks in GUI automation.
Multimodal Large Language Models Suggestion-aware Gradient Relative Policy Optimization pre-operative critic mechanism reasoning-bootstrapping GUI automation GUI-Critic-R1 GUI-Critic-Test GUI-Critic-Train