InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to
  Deliberative Reasoners
Multimodal Large Language Models (MLLMs) have powered Graphical User
Interface (GUI) Agents, showing promise in automating tasks on computing
devices. Recent works have begun exploring reasoning in GUI tasks with
encouraging results. However, many current approaches rely on manually designed
reasoning templates, which may result in reasoning that is not sufficiently
robust and adaptive for complex GUI environments. Meanwhile, some existing
agents continue to operate as Reactive Actors, relying primarily on implicit
reasoning that may lack sufficient depth for GUI tasks demanding planning and
error recovery. We argue that advancing these agents requires a shift from
reactive acting towards acting based on deliberate reasoning. To facilitate
this transformation, we introduce InfiGUI-R1, an MLLM-based GUI agent developed
through our Actor2Reasoner framework, a reasoning-centric, two-stage training
approach designed to progressively evolve agents from Reactive Actors to
Deliberative Reasoners. The first stage, Reasoning Injection, focuses on
establishing a basic reasoner. We employ Spatial Reasoning Distillation to
transfer cross-modal spatial reasoning capabilities from teacher models to
MLLMs through trajectories with explicit reasoning steps, enabling models to
integrate GUI visual-spatial information with logical reasoning before action
generation. The second stage, Deliberation Enhancement, refines the basic
reasoner into a deliberative one using Reinforcement Learning. This stage
introduces two approaches: Sub-goal Guidance, which rewards models for
generating accurate intermediate sub-goals, and Error Recovery Scenario
Construction, which creates failure-and-recovery training scenarios from
identified prone-to-error steps. Experimental results show InfiGUI-R1 achieves
strong performance in GUI grounding and trajectory tasks. Resources at
https://github.com/Reallm-Labs/InfiGUI-R1.
6806e9e819a9fa609609fe65
Yuhang Liu
6806e9e819a9fa609609fe66
Pengxiang Li
claimed_verified
2025-04-22T09:51:50.804Z
64245f2c089d5fae56b4549a
https://cdn-avatars.huggingface.co/v1/production/uploads/64245f2c089d5fae56b4549a/qUHFsL9Svwyj5BKpfMtaY.jpeg
Pengxiang Li
user
pengxiang
6806e9e819a9fa609609fe67
Congkai Xie
extracted_pending
2025-04-22T00:59:21.913Z
629084f3a391a907d5e0484e
/avatars/3a1d6d5aa90b3b8372505d13ccf8f2dd.svg
unk
user
xieck13
6806e9e819a9fa609609fe68
Xavier Hu
6806e9e819a9fa609609fe69
Xiaotian Han
claimed_verified
2025-04-22T09:51:27.543Z
650dde4ce14eeb01d42b37a1
https://cdn-avatars.huggingface.co/v1/production/uploads/650dde4ce14eeb01d42b37a1/n5Yv24uofZ2XJjXdYCrKd.png
Xiaotian Han
user
xiaotianhan
6806e9e819a9fa609609fe6a
Shengyu Zhang
6806e9e819a9fa609609fe6b
Hongxia Yang
6806e9e819a9fa609609fe6c
Fei Wu
InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to
  Deliberative Reasoners
Multimodal Large Language Models (MLLMs) have powered Graphical User
Interface (GUI) Agents, showing promise in automating tasks on computing
devices. Recent works have begun exploring reasoning in GUI tasks with
encouraging results. However, many current approaches rely on manually designed
reasoning templates, which may result in reasoning that is not sufficiently
robust and adaptive for complex GUI environments. Meanwhile, some existing
agents continue to operate as Reactive Actors, relying primarily on implicit
reasoning that may lack sufficient depth for GUI tasks demanding planning and
error recovery. We argue that advancing these agents requires a shift from
reactive acting towards acting based on deliberate reasoning. To facilitate
this transformation, we introduce InfiGUI-R1, an MLLM-based GUI agent developed
through our Actor2Reasoner framework, a reasoning-centric, two-stage training
approach designed to progressively evolve agents from Reactive Actors to
Deliberative Reasoners. The first stage, Reasoning Injection, focuses on
establishing a basic reasoner. We employ Spatial Reasoning Distillation to
transfer cross-modal spatial reasoning capabilities from teacher models to
MLLMs through trajectories with explicit reasoning steps, enabling models to
integrate GUI visual-spatial information with logical reasoning before action
generation. The second stage, Deliberation Enhancement, refines the basic
reasoner into a deliberative one using Reinforcement Learning. This stage
introduces two approaches: Sub-goal Guidance, which rewards models for
generating accurate intermediate sub-goals, and Error Recovery Scenario
Construction, which creates failure-and-recovery training scenarios from
identified prone-to-error steps. Experimental results show InfiGUI-R1 achieves
strong performance in GUI grounding and trajectory tasks. Resources at
https://github.com/Reallm-Labs/InfiGUI-R1.
InfiGUI-R1 leverages a reasoning-centric framework to enhance MLLMs for GUI tasks by transitioning them from Reactive Actors to Deliberative Reasoners using Spatial Reasoning Distillation and Reinforcement Learning techniques.
Multimodal Large Language Models MLLMs Graphical User Interface GUI GUI Agents Reactive Actors Deliberative Reasoners Actor2Reasoner Reasoning Injection Spatial Reasoning Distillation Reinforcement Learning Sub-goal Guidance Error Recovery Scenario Construction GUI grounding trajectory tasks