title: Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
summary: Large language models (LLMs) show remarkable potential to act as computer
  agents, enhancing human productivity and software accessibility in multi-modal
  tasks that require planning and reasoning. However, measuring agent performance
  in realistic environments remains a challenge since: (i) most benchmarks are
  limited to specific modalities or domains (e.g. text-only, web navigation, Q&A,
  coding) and (ii) full benchmark evaluations are slow (on order of magnitude of
  days) given the multi-step sequential nature of tasks. To address these
  challenges, we introduce the Windows Agent Arena: a reproducible, general
  environment focusing exclusively on the Windows operating system (OS) where
  agents can operate freely within a real Windows OS and use the same wide range
  of applications, tools, and web browsers available to human users when solving
  tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse
  Windows tasks across representative domains that require agent abilities in
  planning, screen understanding, and tool usage. Our benchmark is scalable and
  can be seamlessly parallelized in Azure for a full benchmark evaluation in as
  little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we
  also introduce a new multi-modal agent, Navi. Our agent achieves a success rate
  of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted
  human. Navi also demonstrates strong performance on another popular web-based
  benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis
  of Navi's performance, and provide insights into the opportunities for future
  research in agent development and data generation using Windows Agent Arena.
    Webpage: https://microsoft.github.io/WindowsAgentArena
    Code: https://github.com/microsoft/WindowsAgentArena
paper_authors._id: 66e3ac6869d1083dd958c118
paper_authors.name: Rogerio Bonatti
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-13T07:51:21.211Z
paper_authors._id: 6600b688871b36bf846dd188
paper_authors.avatarUrl: /avatars/8cc5cc42819c8dcacee10ca20ec6c057.svg
paper_authors.fullname: Rogerio Bonatti
paper_authors.type: user
paper_authors.user: rbonatti
paper_authors._id: 66e3ac6869d1083dd958c119
paper_authors.name: Dan Zhao
paper_authors._id: 66e3ac6869d1083dd958c11a
paper_authors.name: Francesco Bonacci
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-13T07:52:25.837Z
paper_authors._id: 65ef2edf6e38bf2ff90af403
paper_authors.avatarUrl: /avatars/d05249606f668d33e1b2a0fc8d676d5d.svg
paper_authors.fullname: Francesco Bonacci
paper_authors.type: user
paper_authors.user: francedot
paper_authors._id: 66e3ac6869d1083dd958c11b
paper_authors.name: Dillon Dupont
paper_authors._id: 66e3ac6869d1083dd958c11c
paper_authors.name: Sara Abdali
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-13T07:52:48.796Z
paper_authors._id: 63eebffdbe5d5900b851bd49
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63eebffdbe5d5900b851bd49/MIrZm3rW9nL2MwrRaDZjK.jpeg
paper_authors.fullname: Sara Abdali
paper_authors.type: user
paper_authors.user: Saraabdali
paper_authors._id: 66e3ac6869d1083dd958c11d
paper_authors.name: Yinheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-13T07:52:56.140Z
paper_authors._id: 622a8544305a46df46e9ba7c
paper_authors.avatarUrl: /avatars/b052b3f50822a4a8f3f86ceea4bffa8e.svg
paper_authors.fullname: Yinheng Li
paper_authors.type: user
paper_authors.user: Inhenn
paper_authors._id: 66e3ac6869d1083dd958c11e
paper_authors.name: Justin Wagle
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-13T07:53:03.505Z
paper_authors._id: 6362a1344d4a190d6b14b7b4
paper_authors.avatarUrl: /avatars/fa39fbcfff6ae16ddb0a7144463bfe83.svg
paper_authors.fullname: Justin Wagle
paper_authors.type: user
paper_authors.user: hrz
paper_authors._id: 66e3ac6869d1083dd958c11f
paper_authors.name: Kazuhito Koishida
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-13T07:53:10.281Z
paper_authors._id: 633f6bb8ac3954492880415f
paper_authors.avatarUrl: /avatars/f3dd763d8e092f68cbc33469f148b337.svg
paper_authors.fullname: Kazuhito Koishida
paper_authors.type: user
paper_authors.user: koishida
paper_authors._id: 66e3ac6869d1083dd958c120
paper_authors.name: Arthur Bucker
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-13T07:53:16.472Z
paper_authors._id: 654a9df87f0fd3f2548eba4c
paper_authors.avatarUrl: /avatars/324fbf5daa631331413e7851237f2ebd.svg
paper_authors.fullname: Arthur Fender Coelho Bucker
paper_authors.type: user
paper_authors.user: ArthurBucker
paper_authors._id: 66e3ac6869d1083dd958c121
paper_authors.name: Lawrence Jang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-13T07:20:50.603Z
paper_authors._id: 664aebe829eadb3ab4e4ca3f
paper_authors.avatarUrl: /avatars/548d5656e082c8959ac78b883f0805af.svg
paper_authors.fullname: Lawrence Jang
paper_authors.type: user
paper_authors.user: ljang0
paper_authors._id: 66e3ac6869d1083dd958c122
paper_authors.name: Zack Hui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-09T07:23:50.509Z
paper_authors._id: 6668c40d6da8706f8d531b5f
paper_authors.avatarUrl: /avatars/e0f05182e5e2d8fd101c9548276a1181.svg
paper_authors.fullname: zack
paper_authors.type: user
paper_authors.user: zackhatecoding
paper_title: Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
paper_summary: Large language models (LLMs) show remarkable potential to act as computer
  agents, enhancing human productivity and software accessibility in multi-modal
  tasks that require planning and reasoning. However, measuring agent performance
  in realistic environments remains a challenge since: (i) most benchmarks are
  limited to specific modalities or domains (e.g. text-only, web navigation, Q&A,
  coding) and (ii) full benchmark evaluations are slow (on order of magnitude of
  days) given the multi-step sequential nature of tasks. To address these
  challenges, we introduce the Windows Agent Arena: a reproducible, general
  environment focusing exclusively on the Windows operating system (OS) where
  agents can operate freely within a real Windows OS and use the same wide range
  of applications, tools, and web browsers available to human users when solving
  tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse
  Windows tasks across representative domains that require agent abilities in
  planning, screen understanding, and tool usage. Our benchmark is scalable and
  can be seamlessly parallelized in Azure for a full benchmark evaluation in as
  little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we
  also introduce a new multi-modal agent, Navi. Our agent achieves a success rate
  of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted
  human. Navi also demonstrates strong performance on another popular web-based
  benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis
  of Navi's performance, and provide insights into the opportunities for future
  research in agent development and data generation using Windows Agent Arena.
    Webpage: https://microsoft.github.io/WindowsAgentArena
    Code: https://github.com/microsoft/WindowsAgentArena
paper_ai_summary: Windows Agent Arena provides a general, reproducible environment for evaluating multi-modal agent performance on Windows OS tasks, with scalability and parallelizability features.
paper_ai_keywords: LLMs computer agents human productivity software accessibility multi-modal tasks agent performance realistic environments benchmarks modalities domains planning reasoning Windows operating system OSWorld framework screen understanding tool usage scalable parallelizable Azure multi-modal agent Navi success rate Mind2Web quantitative analysis qualitative analysis future research