title: UFO: A UI-Focused Agent for Windows OS Interaction
summary: We introduce UFO, an innovative UI-Focused agent to fulfill user requests
  tailored to applications on Windows OS, harnessing the capabilities of
  GPT-Vision. UFO employs a dual-agent framework to meticulously observe and
  analyze the graphical user interface (GUI) and control information of Windows
  applications. This enables the agent to seamlessly navigate and operate within
  individual applications and across them to fulfill user requests, even when
  spanning multiple applications. The framework incorporates a control
  interaction module, facilitating action grounding without human intervention
  and enabling fully automated execution. Consequently, UFO transforms arduous
  and time-consuming processes into simple tasks achievable solely through
  natural language commands. We conducted testing of UFO across 9 popular Windows
  applications, encompassing a variety of scenarios reflective of users' daily
  usage. The results, derived from both quantitative metrics and real-case
  studies, underscore the superior effectiveness of UFO in fulfilling user
  requests. To the best of our knowledge, UFO stands as the first UI agent
  specifically tailored for task completion within the Windows OS environment.
  The open-source code for UFO is available on https://github.com/microsoft/UFO.
paper_authors._id: 65cc43527faf059c56232845
paper_authors.name: Chaoyun Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-14T08:41:37.271Z
paper_authors._id: 654dbac9938fbf1e696be8aa
paper_authors.avatarUrl: /avatars/b3c4035c48169c1bfb04a439fce3499f.svg
paper_authors.fullname: Chaoyun Zhang
paper_authors.type: user
paper_authors.user: vyokky
paper_authors._id: 65cc43527faf059c56232846
paper_authors.name: Liqun Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T09:20:27.441Z
paper_authors._id: 666933c97bf97e24f7b5266e
paper_authors.avatarUrl: /avatars/283961b37d463a386b08ad33dacca0f4.svg
paper_authors.fullname: Liqun Li
paper_authors.type: user
paper_authors.user: liqul
paper_authors._id: 65cc43527faf059c56232847
paper_authors.name: Shilin He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-14T12:25:41.789Z
paper_authors._id: 62c6df026a092eda1f1ab6e5
paper_authors.avatarUrl: /avatars/d58fff1a157b189ce2617889ef5f6e2f.svg
paper_authors.fullname: Shilin He
paper_authors.type: user
paper_authors.user: shilhe
paper_authors._id: 65cc43527faf059c56232848
paper_authors.name: Xu Zhang
paper_authors._id: 65cc43527faf059c56232849
paper_authors.name: Bo Qiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-14T12:26:15.471Z
paper_authors._id: 64a51d673badec105d75fe57
paper_authors.avatarUrl: /avatars/c0166a66b468de07b101ec4bc690707b.svg
paper_authors.fullname: Bo Qiao（Not In Microsoft）
paper_authors.type: user
paper_authors.user: bo3333
paper_authors._id: 65cc43527faf059c5623284a
paper_authors.name: Si Qin
paper_authors._id: 65cc43527faf059c5623284b
paper_authors.name: Minghua Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-14T12:29:09.566Z
paper_authors._id: 653b52098abd634b83f4fd38
paper_authors.avatarUrl: /avatars/c9c5023b030e94c1b1abfb7a1c1dfaf3.svg
paper_authors.fullname: MingHua Ma
paper_authors.type: user
paper_authors.user: Gezelligheid520
paper_authors._id: 65cc43527faf059c5623284c
paper_authors.name: Yu Kang
paper_authors._id: 65cc43527faf059c5623284d
paper_authors.name: Qingwei Lin
paper_authors._id: 65cc43527faf059c5623284e
paper_authors.name: Saravan Rajmohan
paper_authors._id: 65cc43527faf059c5623284f
paper_authors.name: Dongmei Zhang
paper_authors._id: 65cc43527faf059c56232850
paper_authors.name: Qi Zhang
paper_title: UFO: A UI-Focused Agent for Windows OS Interaction
paper_summary: We introduce UFO, an innovative UI-Focused agent to fulfill user requests
  tailored to applications on Windows OS, harnessing the capabilities of
  GPT-Vision. UFO employs a dual-agent framework to meticulously observe and
  analyze the graphical user interface (GUI) and control information of Windows
  applications. This enables the agent to seamlessly navigate and operate within
  individual applications and across them to fulfill user requests, even when
  spanning multiple applications. The framework incorporates a control
  interaction module, facilitating action grounding without human intervention
  and enabling fully automated execution. Consequently, UFO transforms arduous
  and time-consuming processes into simple tasks achievable solely through
  natural language commands. We conducted testing of UFO across 9 popular Windows
  applications, encompassing a variety of scenarios reflective of users' daily
  usage. The results, derived from both quantitative metrics and real-case
  studies, underscore the superior effectiveness of UFO in fulfilling user
  requests. To the best of our knowledge, UFO stands as the first UI agent
  specifically tailored for task completion within the Windows OS environment.
  The open-source code for UFO is available on https://github.com/microsoft/UFO.
paper_ai_summary: UFO is a UI-focused agent using GPT-Vision to automate tasks across Windows applications with natural language commands.
paper_ai_keywords: GPT-Vision dual-agent framework graphical user interface control interaction module