UFO2: The Desktop AgentOS
Recent Computer-Using Agents (CUAs), powered by multimodal large language
models (LLMs), offer a promising direction for automating complex desktop
workflows through natural language. However, most existing CUAs remain
conceptual prototypes, hindered by shallow OS integration, fragile
screenshot-based interaction, and disruptive execution.
  We present UFO2, a multiagent AgentOS for Windows desktops that elevates CUAs
into practical, system-level automation. UFO2 features a centralized HostAgent
for task decomposition and coordination, alongside a collection of
application-specialized AppAgent equipped with native APIs, domain-specific
knowledge, and a unified GUI--API action layer. This architecture enables
robust task execution while preserving modularity and extensibility. A hybrid
control detection pipeline fuses Windows UI Automation (UIA) with vision-based
parsing to support diverse interface styles. Runtime efficiency is further
enhanced through speculative multi-action planning, reducing per-step LLM
overhead. Finally, a Picture-in-Picture (PiP) interface enables automation
within an isolated virtual desktop, allowing agents and users to operate
concurrently without interference.
  We evaluate UFO2 across over 20 real-world Windows applications,
demonstrating substantial improvements in robustness and execution accuracy
over prior CUAs. Our results show that deep OS integration unlocks a scalable
path toward reliable, user-aligned desktop automation.
6806fab1b89f3c89c81afbe0
Chaoyun Zhang
claimed_verified
2025-04-22T09:51:25.217Z
654dbac9938fbf1e696be8aa
/avatars/b3c4035c48169c1bfb04a439fce3499f.svg
Chaoyun Zhang
user
vyokky
6806fab1b89f3c89c81afbe1
He Huang
6806fab1b89f3c89c81afbe2
Chiming Ni
6806fab1b89f3c89c81afbe3
Jian Mu
6806fab1b89f3c89c81afbe4
Si Qin
6806fab1b89f3c89c81afbe5
Shilin He
admin_assigned
2025-04-22T10:57:38.634Z
62c6df026a092eda1f1ab6e5
/avatars/d58fff1a157b189ce2617889ef5f6e2f.svg
Shilin He
user
shilhe
6806fab1b89f3c89c81afbe6
Lu Wang
6806fab1b89f3c89c81afbe7
Fangkai Yang
6806fab1b89f3c89c81afbe8
Pu Zhao
6806fab1b89f3c89c81afbe9
Chao Du
admin_assigned
2025-04-22T10:57:25.179Z
632407c892e07e3ca20aca28
/avatars/23b51b37b12b51a0947f687d1de4d3b5.svg
Chao Du
user
duchao
6806fab1b89f3c89c81afbea
Liqun Li
admin_assigned
2025-04-22T10:57:10.526Z
666933c97bf97e24f7b5266e
/avatars/283961b37d463a386b08ad33dacca0f4.svg
Liqun Li
user
liqul
6806fab1b89f3c89c81afbeb
Yu Kang
6806fab1b89f3c89c81afbec
Zhao Jiang
6806fab1b89f3c89c81afbed
Suzhen Zheng
6806fab1b89f3c89c81afbee
Rujia Wang
admin_assigned
2025-04-22T10:56:40.486Z
6430e3d1cd31d174a9f9f097
/avatars/132d28d4336ed93b676123f3fb15bd61.svg
rujiawang
user
rujiawang
6806fab1b89f3c89c81afbef
Jiaxu Qian
6806fab1b89f3c89c81afbf0
Minghua Ma
admin_assigned
2025-04-22T10:56:25.955Z
653b52098abd634b83f4fd38
/avatars/c9c5023b030e94c1b1abfb7a1c1dfaf3.svg
MingHua Ma
user
Gezelligheid520
6806fab1b89f3c89c81afbf1
Jian-Guang Lou
admin_assigned
2025-04-22T10:56:18.953Z
6554338ba4de813482e91ad5
/avatars/bee879fa4a818fd6c070ee001769ce05.svg
Jian-Guang Lou
user
substill
6806fab1b89f3c89c81afbf2
Qingwei Lin
6806fab1b89f3c89c81afbf3
Saravan Rajmohan
6806fab1b89f3c89c81afbf4
Dongmei Zhang
admin_assigned
2025-04-22T10:56:01.402Z
66473d2c7abe6ad66e81a3dd
/avatars/82f40244806c06ffeaa1c4265e9725ea.svg
ZHANGDONGMEI
user
ZDM6426
UFO2: The Desktop AgentOS
Recent Computer-Using Agents (CUAs), powered by multimodal large language
models (LLMs), offer a promising direction for automating complex desktop
workflows through natural language. However, most existing CUAs remain
conceptual prototypes, hindered by shallow OS integration, fragile
screenshot-based interaction, and disruptive execution.
  We present UFO2, a multiagent AgentOS for Windows desktops that elevates CUAs
into practical, system-level automation. UFO2 features a centralized HostAgent
for task decomposition and coordination, alongside a collection of
application-specialized AppAgent equipped with native APIs, domain-specific
knowledge, and a unified GUI--API action layer. This architecture enables
robust task execution while preserving modularity and extensibility. A hybrid
control detection pipeline fuses Windows UI Automation (UIA) with vision-based
parsing to support diverse interface styles. Runtime efficiency is further
enhanced through speculative multi-action planning, reducing per-step LLM
overhead. Finally, a Picture-in-Picture (PiP) interface enables automation
within an isolated virtual desktop, allowing agents and users to operate
concurrently without interference.
  We evaluate UFO2 across over 20 real-world Windows applications,
demonstrating substantial improvements in robustness and execution accuracy
over prior CUAs. Our results show that deep OS integration unlocks a scalable
path toward reliable, user-aligned desktop automation.
UFO2, an AgentOS for Windows, integrates multimodal LLMs with native APIs and hybrid control detection to enhance robust and scalable desktop automation.
multimodal large language models AgentOS HostAgent AppAgent native APIs domain-specific knowledge unified GUI--API action layer Windows UI Automation (UIA) vision-based parsing speculative multi-action planning Picture-in-Picture (PiP)