title: OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real
    Computer Environments
summary: Autonomous agents that accomplish complex computer tasks with minimal human
  interventions have the potential to transform human-computer interaction,
  significantly enhancing accessibility and productivity. However, existing
  benchmarks either lack an interactive environment or are limited to
  environments specific to certain applications or domains, failing to reflect
  the diverse and complex nature of real-world computer use, thereby limiting the
  scope of tasks and agent scalability. To address this issue, we introduce
  OSWorld, the first-of-its-kind scalable, real computer environment for
  multimodal agents, supporting task setup, execution-based evaluation, and
  interactive learning across various operating systems such as Ubuntu, Windows,
  and macOS. OSWorld can serve as a unified, integrated computer environment for
  assessing open-ended computer tasks that involve arbitrary applications.
  Building upon OSWorld, we create a benchmark of 369 computer tasks involving
  real web and desktop apps in open domains, OS file I/O, and workflows spanning
  multiple applications. Each task example is derived from real-world computer
  use cases and includes a detailed initial state setup configuration and a
  custom execution-based evaluation script for reliable, reproducible evaluation.
  Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld
  reveals significant deficiencies in their ability to serve as computer
  assistants. While humans can accomplish over 72.36% of the tasks, the best
  model achieves only 12.24% success, primarily struggling with GUI grounding and
  operational knowledge. Comprehensive analysis using OSWorld provides valuable
  insights for developing multimodal generalist agents that were not possible
  with previous benchmarks. Our code, environment, baseline models, and data are
  publicly available at https://os-world.github.io.
paper_authors._id: 66188c6be1abd2f76dab9661
paper_authors.name: Tianbao Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-12T03:05:11.439Z
paper_authors._id: 618767e4238063b4615d042b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1636263880877-noauth.jpeg
paper_authors.fullname: Tianbao Xie
paper_authors.type: user
paper_authors.user: tianbaoxiexxx
paper_authors._id: 66188c6be1abd2f76dab9662
paper_authors.name: Danyang Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-12T14:49:49.941Z
paper_authors._id: 6455ebad654d8bccae518aaf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/qTvoGqVBuQavfcgbwIoQq.jpeg
paper_authors.fullname: Danyang Zhang
paper_authors.type: user
paper_authors.user: zdy023
paper_authors._id: 66188c6be1abd2f76dab9663
paper_authors.name: Jixuan Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-18T13:15:48.663Z
paper_authors._id: 6465941d0e6c7618f615675b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6465941d0e6c7618f615675b/W4EHqlCucz_bojFLFEeV_.jpeg
paper_authors.fullname: Jixuan Chen
paper_authors.type: user
paper_authors.user: Mayome
paper_authors._id: 66188c6be1abd2f76dab9664
paper_authors.name: Xiaochuan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-12T14:50:14.081Z
paper_authors._id: 64b103cf372d434077206750
paper_authors.avatarUrl: /avatars/ba0eb4fc712a8b9b93ceb30d11859ec2.svg
paper_authors.fullname: Xiaochuan Li
paper_authors.type: user
paper_authors.user: lixiaochuan2020
paper_authors._id: 66188c6be1abd2f76dab9665
paper_authors.name: Siheng Zhao
paper_authors._id: 66188c6be1abd2f76dab9666
paper_authors.name: Ruisheng Cao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-12T14:49:54.777Z
paper_authors._id: 5fbcf0d28f35b82700205fd7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5fbcf0d28f35b82700205fd7/KZPwcaIymBAPs0UHXe5fa.jpeg
paper_authors.fullname: Ruisheng Cao
paper_authors.type: user
paper_authors.user: rshcao
paper_authors._id: 66188c6be1abd2f76dab9667
paper_authors.name: Toh Jing Hua
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-12T07:54:16.193Z
paper_authors._id: 64871af469f5c89076c4d1ef
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/a2x213gR2JSFfzxWyHklT.jpeg
paper_authors.fullname: Toh Jing Hua
paper_authors.type: user
paper_authors.user: jing-hua
paper_authors._id: 66188c6be1abd2f76dab9668
paper_authors.name: Zhoujun Cheng
paper_authors._id: 66188c6be1abd2f76dab9669
paper_authors.name: Dongchan Shin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-11T07:26:26.991Z
paper_authors._id: 63340b24f68a3fb7efa62b3a
paper_authors.avatarUrl: /avatars/44c960437c037553d90b1ca24c952977.svg
paper_authors.fullname: Dongchan Shin
paper_authors.type: user
paper_authors.user: ShinDC
paper_authors._id: 66188c6be1abd2f76dab966a
paper_authors.name: Fangyu Lei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T14:58:47.418Z
paper_authors._id: 64104b467a15af878ae6695d
paper_authors.avatarUrl: /avatars/407983918c12411e5ed636bf7435522b.svg
paper_authors.fullname: Fangyu Lei
paper_authors.type: user
paper_authors.user: FangyuLei
paper_authors._id: 66188c6be1abd2f76dab966b
paper_authors.name: Yitao Liu
paper_authors._id: 66188c6be1abd2f76dab966c
paper_authors.name: Yiheng Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-03T14:25:25.357Z
paper_authors._id: 601d29ab913ad3afd7b7ddb8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1620447944896-601d29ab913ad3afd7b7ddb8.jpeg
paper_authors.fullname: Yiheng Xu
paper_authors.type: user
paper_authors.user: ranpox
paper_authors._id: 66188c6be1abd2f76dab966d
paper_authors.name: Shuyan Zhou
paper_authors._id: 66188c6be1abd2f76dab966e
paper_authors.name: Silvio Savarese
paper_authors._id: 66188c6be1abd2f76dab966f
paper_authors.name: Caiming Xiong
paper_authors._id: 66188c6be1abd2f76dab9670
paper_authors.name: Victor Zhong
paper_authors._id: 66188c6be1abd2f76dab9671
paper_authors.name: Tao Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-28T19:30:43.725Z
paper_authors._id: 629b4c93ba19b79d354270f0
paper_authors.avatarUrl: /avatars/06b8aae4ce030f4e2036d68c08cbcf39.svg
paper_authors.fullname: Tao Yu
paper_authors.type: user
paper_authors.user: taoyds
paper_title: OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real
    Computer Environments
paper_summary: Autonomous agents that accomplish complex computer tasks with minimal human
  interventions have the potential to transform human-computer interaction,
  significantly enhancing accessibility and productivity. However, existing
  benchmarks either lack an interactive environment or are limited to
  environments specific to certain applications or domains, failing to reflect
  the diverse and complex nature of real-world computer use, thereby limiting the
  scope of tasks and agent scalability. To address this issue, we introduce
  OSWorld, the first-of-its-kind scalable, real computer environment for
  multimodal agents, supporting task setup, execution-based evaluation, and
  interactive learning across various operating systems such as Ubuntu, Windows,
  and macOS. OSWorld can serve as a unified, integrated computer environment for
  assessing open-ended computer tasks that involve arbitrary applications.
  Building upon OSWorld, we create a benchmark of 369 computer tasks involving
  real web and desktop apps in open domains, OS file I/O, and workflows spanning
  multiple applications. Each task example is derived from real-world computer
  use cases and includes a detailed initial state setup configuration and a
  custom execution-based evaluation script for reliable, reproducible evaluation.
  Extensive evaluation of state-of-the-art LLM/VLM-based agents on OSWorld
  reveals significant deficiencies in their ability to serve as computer
  assistants. While humans can accomplish over 72.36% of the tasks, the best
  model achieves only 12.24% success, primarily struggling with GUI grounding and
  operational knowledge. Comprehensive analysis using OSWorld provides valuable
  insights for developing multimodal generalist agents that were not possible
  with previous benchmarks. Our code, environment, baseline models, and data are
  publicly available at https://os-world.github.io.
paper_ai_summary: OSWorld provides a scalable, real computer environment for evaluating multimodal agents across various operating systems and tasks, highlighting significant gaps in current LLM/VLM capabilities.
paper_ai_keywords: OSWorld multimodal agents interactive learning GUI grounding execution-based evaluation benchmark LLM VLM operational knowledge