Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal
  Skills
Recent efforts to leverage the Multi-modal Large Language Model (MLLM) as GUI
agents have yielded promising outcomes. However, these agents still struggle
with long-horizon tasks in online environments, primarily due to insufficient
knowledge and the inherent gap between offline and online domains. In this
paper, inspired by how humans generalize knowledge in open-ended environments,
we propose a Hierarchical Multimodal Skills (HMS) module to tackle the issue of
insufficient knowledge. It progressively abstracts trajectories into execution
skills, core skills, and ultimately meta-skills, providing a hierarchical
knowledge structure for long-horizon task planning. To bridge the domain gap,
we propose the Skill-Augmented Monte Carlo Tree Search (SA-MCTS) algorithm,
which efficiently leverages skills acquired in offline environments to reduce
the action search space during online tree exploration. Building on HMS, we
propose Mirage-1, a multimodal, cross-platform, plug-and-play GUI agent. To
validate the performance of Mirage-1 in real-world long-horizon scenarios, we
constructed a new benchmark, AndroidLH. Experimental results show that Mirage-1
outperforms previous agents by 32\%, 19\%, 15\%, and 79\% on AndroidWorld,
MobileMiniWob++, Mind2Web-Live, and AndroidLH, respectively. Project page:
https://cybertronagent.github.io/Mirage-1.github.io/
6850415f2932e11c891b5848
Yuquan Xie
6850415f2932e11c891b5849
Zaijing Li
claimed_verified
2025-06-17T07:22:18.517Z
66b45fe75d0ac130d7d82764
/avatars/09253f41f82c533b36199f82620cd075.svg
Zaijing Li
user
dawn0815
6850415f2932e11c891b584a
Rui Shao
6850415f2932e11c891b584b
Gongwei Chen
6850415f2932e11c891b584c
Kaiwen Zhou
6850415f2932e11c891b584d
Yinchuan Li
6850415f2932e11c891b584e
Dongmei Jiang
6850415f2932e11c891b584f
Liqiang Nie
Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal
  Skills
Recent efforts to leverage the Multi-modal Large Language Model (MLLM) as GUI
agents have yielded promising outcomes. However, these agents still struggle
with long-horizon tasks in online environments, primarily due to insufficient
knowledge and the inherent gap between offline and online domains. In this
paper, inspired by how humans generalize knowledge in open-ended environments,
we propose a Hierarchical Multimodal Skills (HMS) module to tackle the issue of
insufficient knowledge. It progressively abstracts trajectories into execution
skills, core skills, and ultimately meta-skills, providing a hierarchical
knowledge structure for long-horizon task planning. To bridge the domain gap,
we propose the Skill-Augmented Monte Carlo Tree Search (SA-MCTS) algorithm,
which efficiently leverages skills acquired in offline environments to reduce
the action search space during online tree exploration. Building on HMS, we
propose Mirage-1, a multimodal, cross-platform, plug-and-play GUI agent. To
validate the performance of Mirage-1 in real-world long-horizon scenarios, we
constructed a new benchmark, AndroidLH. Experimental results show that Mirage-1
outperforms previous agents by 32\%, 19\%, 15\%, and 79\% on AndroidWorld,
MobileMiniWob++, Mind2Web-Live, and AndroidLH, respectively. Project page:
https://cybertronagent.github.io/Mirage-1.github.io/
Hierarchical Multimodal Skills and Skill-Augmented Monte Carlo Tree Search improve multimodal GUI agent performance in long-horizon tasks by abstracting knowledge and bridging the offline-online domain gap.
Hierarchical Multimodal Skills Skill-Augmented Monte Carlo Tree Search GUI agents long-horizon tasks offline environments online environments execution skills core skills meta-skills action search space knowledge structure task planning AndroidLH benchmark Mirage-1 AndroidWorld MobileMiniWob++ Mind2Web-Live