UFO: A UI-Focused Agent for Windows OS Interaction
We introduce UFO, an innovative UI-Focused agent to fulfill user requests
tailored to applications on Windows OS, harnessing the capabilities of
GPT-Vision. UFO employs a dual-agent framework to meticulously observe and
analyze the graphical user interface (GUI) and control information of Windows
applications. This enables the agent to seamlessly navigate and operate within
individual applications and across them to fulfill user requests, even when
spanning multiple applications. The framework incorporates a control
interaction module, facilitating action grounding without human intervention
and enabling fully automated execution. Consequently, UFO transforms arduous
and time-consuming processes into simple tasks achievable solely through
natural language commands. We conducted testing of UFO across 9 popular Windows
applications, encompassing a variety of scenarios reflective of users' daily
usage. The results, derived from both quantitative metrics and real-case
studies, underscore the superior effectiveness of UFO in fulfilling user
requests. To the best of our knowledge, UFO stands as the first UI agent
specifically tailored for task completion within the Windows OS environment.
The open-source code for UFO is available on https://github.com/microsoft/UFO.
65cc43527faf059c56232845
Chaoyun Zhang
claimed_verified
2024-02-14T08:41:37.271Z
654dbac9938fbf1e696be8aa
/avatars/b3c4035c48169c1bfb04a439fce3499f.svg
Chaoyun Zhang
user
vyokky
65cc43527faf059c56232846
Liqun Li
claimed_verified
2024-11-28T09:20:27.441Z
666933c97bf97e24f7b5266e
/avatars/283961b37d463a386b08ad33dacca0f4.svg
Liqun Li
user
liqul
65cc43527faf059c56232847
Shilin He
admin_assigned
2024-02-14T12:25:41.789Z
62c6df026a092eda1f1ab6e5
/avatars/d58fff1a157b189ce2617889ef5f6e2f.svg
Shilin He
user
shilhe
65cc43527faf059c56232848
Xu Zhang
65cc43527faf059c56232849
Bo Qiao
admin_assigned
2024-02-14T12:26:15.471Z
64a51d673badec105d75fe57
/avatars/c0166a66b468de07b101ec4bc690707b.svg
Bo Qiao（Not In Microsoft）
user
bo3333
65cc43527faf059c5623284a
Si Qin
65cc43527faf059c5623284b
Minghua Ma
admin_assigned
2024-02-14T12:29:09.566Z
653b52098abd634b83f4fd38
/avatars/c9c5023b030e94c1b1abfb7a1c1dfaf3.svg
MingHua Ma
user
Gezelligheid520
65cc43527faf059c5623284c
Yu Kang
65cc43527faf059c5623284d
Qingwei Lin
65cc43527faf059c5623284e
Saravan Rajmohan
65cc43527faf059c5623284f
Dongmei Zhang
65cc43527faf059c56232850
Qi Zhang
UFO: A UI-Focused Agent for Windows OS Interaction
We introduce UFO, an innovative UI-Focused agent to fulfill user requests
tailored to applications on Windows OS, harnessing the capabilities of
GPT-Vision. UFO employs a dual-agent framework to meticulously observe and
analyze the graphical user interface (GUI) and control information of Windows
applications. This enables the agent to seamlessly navigate and operate within
individual applications and across them to fulfill user requests, even when
spanning multiple applications. The framework incorporates a control
interaction module, facilitating action grounding without human intervention
and enabling fully automated execution. Consequently, UFO transforms arduous
and time-consuming processes into simple tasks achievable solely through
natural language commands. We conducted testing of UFO across 9 popular Windows
applications, encompassing a variety of scenarios reflective of users' daily
usage. The results, derived from both quantitative metrics and real-case
studies, underscore the superior effectiveness of UFO in fulfilling user
requests. To the best of our knowledge, UFO stands as the first UI agent
specifically tailored for task completion within the Windows OS environment.
The open-source code for UFO is available on https://github.com/microsoft/UFO.
UFO is a UI-focused agent using GPT-Vision to automate tasks across Windows applications with natural language commands.
GPT-Vision dual-agent framework graphical user interface control interaction module