InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning
  and Reflection
Graphical User Interface (GUI) Agents, powered by multimodal large language
models (MLLMs), have shown great potential for task automation on computing
devices such as computers and mobile phones. However, existing agents face
challenges in multi-step reasoning and reliance on textual annotations,
limiting their effectiveness. We introduce InfiGUIAgent, an MLLM-based
GUI Agent trained with a two-stage supervised fine-tuning pipeline. Stage 1
enhances fundamental skills such as GUI understanding and grounding, while
Stage 2 integrates hierarchical reasoning and expectation-reflection reasoning
skills using synthesized data to enable native reasoning abilities of the
agents. InfiGUIAgent achieves competitive performance on several GUI
benchmarks, highlighting the impact of native reasoning skills in enhancing GUI
interaction for automation tasks. Resources are available at
https://github.com/Reallm-Labs/InfiGUIAgent.
677f5089a0843d06966ac68e
Yuhang Liu
admin_assigned
2025-01-09T20:24:38.607Z
66dd4feb14f4776a44b071f2
/avatars/8b06fa3019999abb762de2b8d9961e2b.svg
Yuhang Liu
user
CausalLLMs
677f5089a0843d06966ac68f
Pengxiang Li
claimed_verified
2025-01-09T10:06:37.729Z
64245f2c089d5fae56b4549a
https://cdn-avatars.huggingface.co/v1/production/uploads/64245f2c089d5fae56b4549a/qUHFsL9Svwyj5BKpfMtaY.jpeg
Pengxiang Li
user
pengxiang
677f5089a0843d06966ac690
Zishu Wei
admin_assigned
2025-01-09T20:24:47.683Z
63909970937867f0cb400b09
/avatars/a360d5d5a3605e7312b078312eee79b6.svg
zishu wei
user
shu06
677f5089a0843d06966ac691
Congkai Xie
admin_assigned
2025-01-09T20:24:53.767Z
6719f1ad725123d503b5ef3c
/avatars/08e1be1f4afa1b6e1501a15cdb786a47.svg
Congkai Xie
user
congkai
677f5089a0843d06966ac692
Xueyu Hu
admin_assigned
2025-01-09T20:25:09.450Z
65897684f8b453e1f57cdb26
/avatars/80096d6c808805e1a84a68fb6194a7d4.svg
huxueyu
user
huxueyu
677f5089a0843d06966ac693
Xinchen Xu
admin_assigned
2025-01-09T20:25:15.603Z
65601713baf1f5f902292bb6
/avatars/57c65cff29ffb4f25b943b5baccdc795.svg
Xinchen Xu
user
SHU-Xuxc
677f5089a0843d06966ac694
Shengyu Zhang
677f5089a0843d06966ac695
Xiaotian Han
claimed_verified
2025-01-09T19:57:55.175Z
650dde4ce14eeb01d42b37a1
https://cdn-avatars.huggingface.co/v1/production/uploads/650dde4ce14eeb01d42b37a1/n5Yv24uofZ2XJjXdYCrKd.png
Xiaotian Han
user
xiaotianhan
677f5089a0843d06966ac696
Hongxia Yang
677f5089a0843d06966ac697
Fei Wu
InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning
  and Reflection
Graphical User Interface (GUI) Agents, powered by multimodal large language
models (MLLMs), have shown great potential for task automation on computing
devices such as computers and mobile phones. However, existing agents face
challenges in multi-step reasoning and reliance on textual annotations,
limiting their effectiveness. We introduce InfiGUIAgent, an MLLM-based
GUI Agent trained with a two-stage supervised fine-tuning pipeline. Stage 1
enhances fundamental skills such as GUI understanding and grounding, while
Stage 2 integrates hierarchical reasoning and expectation-reflection reasoning
skills using synthesized data to enable native reasoning abilities of the
agents. InfiGUIAgent achieves competitive performance on several GUI
benchmarks, highlighting the impact of native reasoning skills in enhancing GUI
interaction for automation tasks. Resources are available at
https://github.com/Reallm-Labs/InfiGUIAgent.
InfiGUIAgent, an MLLM-based GUI Agent, enhances multi-step reasoning and GUI interaction through a two-stage supervised fine-tuning pipeline.
multimodal large language models GUI Agents supervised fine-tuning GUI understanding hierarchical reasoning expectation-reflection reasoning synthesized data GUI benchmarks