title: UFO2: The Desktop AgentOS
summary: Recent Computer-Using Agents (CUAs), powered by multimodal large language
  models (LLMs), offer a promising direction for automating complex desktop
  workflows through natural language. However, most existing CUAs remain
  conceptual prototypes, hindered by shallow OS integration, fragile
  screenshot-based interaction, and disruptive execution.
    We present UFO2, a multiagent AgentOS for Windows desktops that elevates CUAs
  into practical, system-level automation. UFO2 features a centralized HostAgent
  for task decomposition and coordination, alongside a collection of
  application-specialized AppAgent equipped with native APIs, domain-specific
  knowledge, and a unified GUI--API action layer. This architecture enables
  robust task execution while preserving modularity and extensibility. A hybrid
  control detection pipeline fuses Windows UI Automation (UIA) with vision-based
  parsing to support diverse interface styles. Runtime efficiency is further
  enhanced through speculative multi-action planning, reducing per-step LLM
  overhead. Finally, a Picture-in-Picture (PiP) interface enables automation
  within an isolated virtual desktop, allowing agents and users to operate
  concurrently without interference.
    We evaluate UFO2 across over 20 real-world Windows applications,
  demonstrating substantial improvements in robustness and execution accuracy
  over prior CUAs. Our results show that deep OS integration unlocks a scalable
  path toward reliable, user-aligned desktop automation.
paper_authors._id: 6806fab1b89f3c89c81afbe0
paper_authors.name: Chaoyun Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T09:51:25.217Z
paper_authors._id: 654dbac9938fbf1e696be8aa
paper_authors.avatarUrl: /avatars/b3c4035c48169c1bfb04a439fce3499f.svg
paper_authors.fullname: Chaoyun Zhang
paper_authors.type: user
paper_authors.user: vyokky
paper_authors._id: 6806fab1b89f3c89c81afbe1
paper_authors.name: He Huang
paper_authors._id: 6806fab1b89f3c89c81afbe2
paper_authors.name: Chiming Ni
paper_authors._id: 6806fab1b89f3c89c81afbe3
paper_authors.name: Jian Mu
paper_authors._id: 6806fab1b89f3c89c81afbe4
paper_authors.name: Si Qin
paper_authors._id: 6806fab1b89f3c89c81afbe5
paper_authors.name: Shilin He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T10:57:38.634Z
paper_authors._id: 62c6df026a092eda1f1ab6e5
paper_authors.avatarUrl: /avatars/d58fff1a157b189ce2617889ef5f6e2f.svg
paper_authors.fullname: Shilin He
paper_authors.type: user
paper_authors.user: shilhe
paper_authors._id: 6806fab1b89f3c89c81afbe6
paper_authors.name: Lu Wang
paper_authors._id: 6806fab1b89f3c89c81afbe7
paper_authors.name: Fangkai Yang
paper_authors._id: 6806fab1b89f3c89c81afbe8
paper_authors.name: Pu Zhao
paper_authors._id: 6806fab1b89f3c89c81afbe9
paper_authors.name: Chao Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T10:57:25.179Z
paper_authors._id: 632407c892e07e3ca20aca28
paper_authors.avatarUrl: /avatars/23b51b37b12b51a0947f687d1de4d3b5.svg
paper_authors.fullname: Chao Du
paper_authors.type: user
paper_authors.user: duchao
paper_authors._id: 6806fab1b89f3c89c81afbea
paper_authors.name: Liqun Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T10:57:10.526Z
paper_authors._id: 666933c97bf97e24f7b5266e
paper_authors.avatarUrl: /avatars/283961b37d463a386b08ad33dacca0f4.svg
paper_authors.fullname: Liqun Li
paper_authors.type: user
paper_authors.user: liqul
paper_authors._id: 6806fab1b89f3c89c81afbeb
paper_authors.name: Yu Kang
paper_authors._id: 6806fab1b89f3c89c81afbec
paper_authors.name: Zhao Jiang
paper_authors._id: 6806fab1b89f3c89c81afbed
paper_authors.name: Suzhen Zheng
paper_authors._id: 6806fab1b89f3c89c81afbee
paper_authors.name: Rujia Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T10:56:40.486Z
paper_authors._id: 6430e3d1cd31d174a9f9f097
paper_authors.avatarUrl: /avatars/132d28d4336ed93b676123f3fb15bd61.svg
paper_authors.fullname: rujiawang
paper_authors.type: user
paper_authors.user: rujiawang
paper_authors._id: 6806fab1b89f3c89c81afbef
paper_authors.name: Jiaxu Qian
paper_authors._id: 6806fab1b89f3c89c81afbf0
paper_authors.name: Minghua Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T10:56:25.955Z
paper_authors._id: 653b52098abd634b83f4fd38
paper_authors.avatarUrl: /avatars/c9c5023b030e94c1b1abfb7a1c1dfaf3.svg
paper_authors.fullname: MingHua Ma
paper_authors.type: user
paper_authors.user: Gezelligheid520
paper_authors._id: 6806fab1b89f3c89c81afbf1
paper_authors.name: Jian-Guang Lou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T10:56:18.953Z
paper_authors._id: 6554338ba4de813482e91ad5
paper_authors.avatarUrl: /avatars/bee879fa4a818fd6c070ee001769ce05.svg
paper_authors.fullname: Jian-Guang Lou
paper_authors.type: user
paper_authors.user: substill
paper_authors._id: 6806fab1b89f3c89c81afbf2
paper_authors.name: Qingwei Lin
paper_authors._id: 6806fab1b89f3c89c81afbf3
paper_authors.name: Saravan Rajmohan
paper_authors._id: 6806fab1b89f3c89c81afbf4
paper_authors.name: Dongmei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T10:56:01.402Z
paper_authors._id: 66473d2c7abe6ad66e81a3dd
paper_authors.avatarUrl: /avatars/82f40244806c06ffeaa1c4265e9725ea.svg
paper_authors.fullname: ZHANGDONGMEI
paper_authors.type: user
paper_authors.user: ZDM6426
paper_title: UFO2: The Desktop AgentOS
paper_summary: Recent Computer-Using Agents (CUAs), powered by multimodal large language
  models (LLMs), offer a promising direction for automating complex desktop
  workflows through natural language. However, most existing CUAs remain
  conceptual prototypes, hindered by shallow OS integration, fragile
  screenshot-based interaction, and disruptive execution.
    We present UFO2, a multiagent AgentOS for Windows desktops that elevates CUAs
  into practical, system-level automation. UFO2 features a centralized HostAgent
  for task decomposition and coordination, alongside a collection of
  application-specialized AppAgent equipped with native APIs, domain-specific
  knowledge, and a unified GUI--API action layer. This architecture enables
  robust task execution while preserving modularity and extensibility. A hybrid
  control detection pipeline fuses Windows UI Automation (UIA) with vision-based
  parsing to support diverse interface styles. Runtime efficiency is further
  enhanced through speculative multi-action planning, reducing per-step LLM
  overhead. Finally, a Picture-in-Picture (PiP) interface enables automation
  within an isolated virtual desktop, allowing agents and users to operate
  concurrently without interference.
    We evaluate UFO2 across over 20 real-world Windows applications,
  demonstrating substantial improvements in robustness and execution accuracy
  over prior CUAs. Our results show that deep OS integration unlocks a scalable
  path toward reliable, user-aligned desktop automation.
paper_ai_summary: UFO2, an AgentOS for Windows, integrates multimodal LLMs with native APIs and hybrid control detection to enhance robust and scalable desktop automation.
paper_ai_keywords: multimodal large language models AgentOS HostAgent AppAgent native APIs domain-specific knowledge unified GUI--API action layer Windows UI Automation (UIA) vision-based parsing speculative multi-action planning Picture-in-Picture (PiP)