title: InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning
    and Reflection
summary: Graphical User Interface (GUI) Agents, powered by multimodal large language
  models (MLLMs), have shown great potential for task automation on computing
  devices such as computers and mobile phones. However, existing agents face
  challenges in multi-step reasoning and reliance on textual annotations,
  limiting their effectiveness. We introduce InfiGUIAgent, an MLLM-based
  GUI Agent trained with a two-stage supervised fine-tuning pipeline. Stage 1
  enhances fundamental skills such as GUI understanding and grounding, while
  Stage 2 integrates hierarchical reasoning and expectation-reflection reasoning
  skills using synthesized data to enable native reasoning abilities of the
  agents. InfiGUIAgent achieves competitive performance on several GUI
  benchmarks, highlighting the impact of native reasoning skills in enhancing GUI
  interaction for automation tasks. Resources are available at
  https://github.com/Reallm-Labs/InfiGUIAgent.
paper_authors._id: 677f5089a0843d06966ac68e
paper_authors.name: Yuhang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-09T20:24:38.607Z
paper_authors._id: 66dd4feb14f4776a44b071f2
paper_authors.avatarUrl: /avatars/8b06fa3019999abb762de2b8d9961e2b.svg
paper_authors.fullname: Yuhang Liu
paper_authors.type: user
paper_authors.user: CausalLLMs
paper_authors._id: 677f5089a0843d06966ac68f
paper_authors.name: Pengxiang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-09T10:06:37.729Z
paper_authors._id: 64245f2c089d5fae56b4549a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64245f2c089d5fae56b4549a/qUHFsL9Svwyj5BKpfMtaY.jpeg
paper_authors.fullname: Pengxiang Li
paper_authors.type: user
paper_authors.user: pengxiang
paper_authors._id: 677f5089a0843d06966ac690
paper_authors.name: Zishu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-09T20:24:47.683Z
paper_authors._id: 63909970937867f0cb400b09
paper_authors.avatarUrl: /avatars/a360d5d5a3605e7312b078312eee79b6.svg
paper_authors.fullname: zishu wei
paper_authors.type: user
paper_authors.user: shu06
paper_authors._id: 677f5089a0843d06966ac691
paper_authors.name: Congkai Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-09T20:24:53.767Z
paper_authors._id: 6719f1ad725123d503b5ef3c
paper_authors.avatarUrl: /avatars/08e1be1f4afa1b6e1501a15cdb786a47.svg
paper_authors.fullname: Congkai Xie
paper_authors.type: user
paper_authors.user: congkai
paper_authors._id: 677f5089a0843d06966ac692
paper_authors.name: Xueyu Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-09T20:25:09.450Z
paper_authors._id: 65897684f8b453e1f57cdb26
paper_authors.avatarUrl: /avatars/80096d6c808805e1a84a68fb6194a7d4.svg
paper_authors.fullname: huxueyu
paper_authors.type: user
paper_authors.user: huxueyu
paper_authors._id: 677f5089a0843d06966ac693
paper_authors.name: Xinchen Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-09T20:25:15.603Z
paper_authors._id: 65601713baf1f5f902292bb6
paper_authors.avatarUrl: /avatars/57c65cff29ffb4f25b943b5baccdc795.svg
paper_authors.fullname: Xinchen Xu
paper_authors.type: user
paper_authors.user: SHU-Xuxc
paper_authors._id: 677f5089a0843d06966ac694
paper_authors.name: Shengyu Zhang
paper_authors._id: 677f5089a0843d06966ac695
paper_authors.name: Xiaotian Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-09T19:57:55.175Z
paper_authors._id: 650dde4ce14eeb01d42b37a1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/650dde4ce14eeb01d42b37a1/n5Yv24uofZ2XJjXdYCrKd.png
paper_authors.fullname: Xiaotian Han
paper_authors.type: user
paper_authors.user: xiaotianhan
paper_authors._id: 677f5089a0843d06966ac696
paper_authors.name: Hongxia Yang
paper_authors._id: 677f5089a0843d06966ac697
paper_authors.name: Fei Wu
paper_title: InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning
    and Reflection
paper_summary: Graphical User Interface (GUI) Agents, powered by multimodal large language
  models (MLLMs), have shown great potential for task automation on computing
  devices such as computers and mobile phones. However, existing agents face
  challenges in multi-step reasoning and reliance on textual annotations,
  limiting their effectiveness. We introduce InfiGUIAgent, an MLLM-based
  GUI Agent trained with a two-stage supervised fine-tuning pipeline. Stage 1
  enhances fundamental skills such as GUI understanding and grounding, while
  Stage 2 integrates hierarchical reasoning and expectation-reflection reasoning
  skills using synthesized data to enable native reasoning abilities of the
  agents. InfiGUIAgent achieves competitive performance on several GUI
  benchmarks, highlighting the impact of native reasoning skills in enhancing GUI
  interaction for automation tasks. Resources are available at
  https://github.com/Reallm-Labs/InfiGUIAgent.
paper_ai_summary: InfiGUIAgent, an MLLM-based GUI Agent, enhances multi-step reasoning and GUI interaction through a two-stage supervised fine-tuning pipeline.
paper_ai_keywords: multimodal large language models GUI Agents supervised fine-tuning GUI understanding hierarchical reasoning expectation-reflection reasoning synthesized data GUI benchmarks