UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based
  Mobile GUI Agents
In this paper, we introduce UI-Genie, a self-improving framework addressing
two key challenges in GUI agents: verification of trajectory outcome is
challenging and high-quality training data are not scalable. These challenges
are addressed by a reward model and a self-improving pipeline, respectively.
The reward model, UI-Genie-RM, features an image-text interleaved architecture
that efficiently pro- cesses historical context and unifies action-level and
task-level rewards. To sup- port the training of UI-Genie-RM, we develop
deliberately-designed data genera- tion strategies including rule-based
verification, controlled trajectory corruption, and hard negative mining. To
address the second challenge, a self-improvement pipeline progressively expands
solvable complex GUI tasks by enhancing both the agent and reward models
through reward-guided exploration and outcome verification in dynamic
environments. For training the model, we generate UI- Genie-RM-517k and
UI-Genie-Agent-16k, establishing the first reward-specific dataset for GUI
agents while demonstrating high-quality synthetic trajectory gen- eration
without manual annotation. Experimental results show that UI-Genie achieves
state-of-the-art performance across multiple GUI agent benchmarks with three
generations of data-model self-improvement. We open-source our complete
framework implementation and generated datasets to facilitate further research
in https://github.com/Euphoria16/UI-Genie.
683698f3c32e462c40a9188f
Han Xiao
claimed_verified
2025-05-28T08:56:45.845Z
666aa99cd1652853e4f9a8b9
/avatars/7cd5a0c34b5ccb8eff5a353d88d15a93.svg
HanXiao
user
HanXiao1999
683698f3c32e462c40a91890
Guozhi Wang
admin_assigned
2025-05-28T12:17:24.039Z
65446e3fdc39aa9fabba8fd5
/avatars/41321bd37d007387d1d363b6b4fd2079.svg
guozhi wang
user
juice-wang
683698f3c32e462c40a91891
Yuxiang Chai
admin_assigned
2025-05-28T12:17:29.742Z
6458ce236fa580137af5aa95
/avatars/db65a7332e375eb5daad5c1b076b1e3b.svg
Yuxiang Chai
user
Yuxiang007
683698f3c32e462c40a91892
Zimu Lu
admin_assigned
2025-05-28T12:17:35.790Z
64b0bfef2f2f9c345b87e673
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/7O3zCr_IyGvtz0BUdKNKi.jpeg
Zimu Lu
user
luzimu
683698f3c32e462c40a91893
Weifeng Lin
admin_assigned
2025-05-28T12:17:49.451Z
66026c9068d519ed32519e9c
/avatars/8fa051312c713772e5b8ba65989ff7f5.svg
Weifeng Lin
user
Afeng-x
683698f3c32e462c40a91894
Hao He
683698f3c32e462c40a91895
Lue Fan
683698f3c32e462c40a91896
Liuyang Bian
admin_assigned
2025-05-28T12:17:55.352Z
66d459ca1dbd780574b42e2d
/avatars/d573a92487628bf6c727b77eedf80b9d.svg
bian liuyang
user
liuyangbian
683698f3c32e462c40a91897
Rui Hu
683698f3c32e462c40a91898
Liang Liu
683698f3c32e462c40a91899
Shuai Ren
683698f3c32e462c40a9189a
Yafei Wen
admin_assigned
2025-05-28T12:18:16.701Z
67134b1ce58b557752c54a1c
/avatars/5cc07748dac424a6f5c7ca09f1c45068.svg
yafei wen
user
wolf1110
683698f3c32e462c40a9189b
Xiaoxin Chen
683698f3c32e462c40a9189c
Aojun Zhou
claimed_verified
2025-05-28T08:56:48.292Z
637de1520d5bb06fbe5207a9
/avatars/1090851217270c5a858b13e013356d4f.svg
AJ.Zhou
user
AJZhou
683698f3c32e462c40a9189d
Hongsheng Li
admin_assigned
2025-05-28T12:18:35.337Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based
  Mobile GUI Agents
In this paper, we introduce UI-Genie, a self-improving framework addressing
two key challenges in GUI agents: verification of trajectory outcome is
challenging and high-quality training data are not scalable. These challenges
are addressed by a reward model and a self-improving pipeline, respectively.
The reward model, UI-Genie-RM, features an image-text interleaved architecture
that efficiently pro- cesses historical context and unifies action-level and
task-level rewards. To sup- port the training of UI-Genie-RM, we develop
deliberately-designed data genera- tion strategies including rule-based
verification, controlled trajectory corruption, and hard negative mining. To
address the second challenge, a self-improvement pipeline progressively expands
solvable complex GUI tasks by enhancing both the agent and reward models
through reward-guided exploration and outcome verification in dynamic
environments. For training the model, we generate UI- Genie-RM-517k and
UI-Genie-Agent-16k, establishing the first reward-specific dataset for GUI
agents while demonstrating high-quality synthetic trajectory gen- eration
without manual annotation. Experimental results show that UI-Genie achieves
state-of-the-art performance across multiple GUI agent benchmarks with three
generations of data-model self-improvement. We open-source our complete
framework implementation and generated datasets to facilitate further research
in https://github.com/Euphoria16/UI-Genie.
UI-Genie framework addresses GUI agent challenges through a reward model with image-text architecture and a self-improvement pipeline, achieving state-of-the-art performance on multiple benchmarks.
image-text interleaved architecture GUI agents reward model self-improving pipeline rule-based verification controlled trajectory corruption hard negative mining reward-guided exploration outcome verification dynamic environments synthetic trajectory generation