title: OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
summary: Autonomous interaction with the computer has been a longstanding challenge
  with great potential, and the recent proliferation of large language models
  (LLMs) has markedly accelerated progress in building digital agents. However,
  most of these agents are designed to interact with a narrow domain, such as a
  specific software or website. This narrow focus constrains their applicability
  for general computer tasks. To this end, we introduce OS-Copilot, a framework
  to build generalist agents capable of interfacing with comprehensive elements
  in an operating system (OS), including the web, code terminals, files,
  multimedia, and various third-party applications. We use OS-Copilot to create
  FRIDAY, a self-improving embodied agent for automating general computer tasks.
  On GAIA, a general AI assistants benchmark, FRIDAY outperforms previous methods
  by 35%, showcasing strong generalization to unseen applications via accumulated
  skills from previous tasks. We also present numerical and quantitative evidence
  that FRIDAY learns to control and self-improve on Excel and Powerpoint with
  minimal supervision. Our OS-Copilot framework and empirical findings provide
  infrastructure and insights for future research toward more capable and
  general-purpose computer agents.
paper_authors._id: 65cadc746938a6a81ff24a96
paper_authors.name: Zhiyong Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T08:28:59.225Z
paper_authors._id: 6487e877b541258e038ea921
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/jXU7VV9V7_X530IOOCvVZ.jpeg
paper_authors.fullname: wuzhiyong
paper_authors.type: user
paper_authors.user: zhiyongdezhi
paper_authors._id: 65cadc746938a6a81ff24a97
paper_authors.name: Chengcheng Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T08:29:09.819Z
paper_authors._id: 619ddd708ae9cafd72ab20d5
paper_authors.avatarUrl: /avatars/6b44e4928de0fc27287bf922c3f1802d.svg
paper_authors.fullname: Chengcheng Han
paper_authors.type: user
paper_authors.user: hccngu
paper_authors._id: 65cadc746938a6a81ff24a98
paper_authors.name: Zichen Ding
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-15T09:02:14.697Z
paper_authors._id: 642b9861bb77f8456634b048
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642b9861bb77f8456634b048/ZT-oJrw5BsADC-gZT_i25.jpeg
paper_authors.fullname: Zichen Ding
paper_authors.type: user
paper_authors.user: heroding77
paper_authors._id: 65cadc746938a6a81ff24a99
paper_authors.name: Zhenmin Weng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T08:29:37.101Z
paper_authors._id: 64d7016da146b1c0a67f1b4f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0dw_wxEBtFp2bSQubJO9K.jpeg
paper_authors.fullname: Zhenmin Weng
paper_authors.type: user
paper_authors.user: Tamiflu233
paper_authors._id: 65cadc746938a6a81ff24a9a
paper_authors.name: Zhoumianze Liu
paper_authors._id: 65cadc746938a6a81ff24a9b
paper_authors.name: Shunyu Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T08:29:52.228Z
paper_authors._id: 62aa3b8f46d66bafe2f55899
paper_authors.avatarUrl: /avatars/d45c6406042c3f6167153aba76d03776.svg
paper_authors.fullname: Shunyu Yao
paper_authors.type: user
paper_authors.user: ysymyth
paper_authors._id: 65cadc746938a6a81ff24a9c
paper_authors.name: Tao Yu
paper_authors._id: 65cadc746938a6a81ff24a9d
paper_authors.name: Lingpeng Kong
paper_title: OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
paper_summary: Autonomous interaction with the computer has been a longstanding challenge
  with great potential, and the recent proliferation of large language models
  (LLMs) has markedly accelerated progress in building digital agents. However,
  most of these agents are designed to interact with a narrow domain, such as a
  specific software or website. This narrow focus constrains their applicability
  for general computer tasks. To this end, we introduce OS-Copilot, a framework
  to build generalist agents capable of interfacing with comprehensive elements
  in an operating system (OS), including the web, code terminals, files,
  multimedia, and various third-party applications. We use OS-Copilot to create
  FRIDAY, a self-improving embodied agent for automating general computer tasks.
  On GAIA, a general AI assistants benchmark, FRIDAY outperforms previous methods
  by 35%, showcasing strong generalization to unseen applications via accumulated
  skills from previous tasks. We also present numerical and quantitative evidence
  that FRIDAY learns to control and self-improve on Excel and Powerpoint with
  minimal supervision. Our OS-Copilot framework and empirical findings provide
  infrastructure and insights for future research toward more capable and
  general-purpose computer agents.
paper_ai_summary: OS-Copilot framework enables FRIDAY, a self-improving generalist agent, to automate a wide range of computer tasks with enhanced generalization and minimal supervision.
paper_ai_keywords: LLMs OS-Copilot digital agents embodied agent general AI assistants benchmark GAIA self-improving general computer tasks minimal supervision