OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
Autonomous interaction with the computer has been a longstanding challenge
with great potential, and the recent proliferation of large language models
(LLMs) has markedly accelerated progress in building digital agents. However,
most of these agents are designed to interact with a narrow domain, such as a
specific software or website. This narrow focus constrains their applicability
for general computer tasks. To this end, we introduce OS-Copilot, a framework
to build generalist agents capable of interfacing with comprehensive elements
in an operating system (OS), including the web, code terminals, files,
multimedia, and various third-party applications. We use OS-Copilot to create
FRIDAY, a self-improving embodied agent for automating general computer tasks.
On GAIA, a general AI assistants benchmark, FRIDAY outperforms previous methods
by 35%, showcasing strong generalization to unseen applications via accumulated
skills from previous tasks. We also present numerical and quantitative evidence
that FRIDAY learns to control and self-improve on Excel and Powerpoint with
minimal supervision. Our OS-Copilot framework and empirical findings provide
infrastructure and insights for future research toward more capable and
general-purpose computer agents.
65cadc746938a6a81ff24a96
Zhiyong Wu
admin_assigned
2024-02-13T08:28:59.225Z
6487e877b541258e038ea921
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/jXU7VV9V7_X530IOOCvVZ.jpeg
wuzhiyong
user
zhiyongdezhi
65cadc746938a6a81ff24a97
Chengcheng Han
admin_assigned
2024-02-13T08:29:09.819Z
619ddd708ae9cafd72ab20d5
/avatars/6b44e4928de0fc27287bf922c3f1802d.svg
Chengcheng Han
user
hccngu
65cadc746938a6a81ff24a98
Zichen Ding
claimed_verified
2024-07-15T09:02:14.697Z
642b9861bb77f8456634b048
https://cdn-avatars.huggingface.co/v1/production/uploads/642b9861bb77f8456634b048/ZT-oJrw5BsADC-gZT_i25.jpeg
Zichen Ding
user
heroding77
65cadc746938a6a81ff24a99
Zhenmin Weng
admin_assigned
2024-02-13T08:29:37.101Z
64d7016da146b1c0a67f1b4f
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0dw_wxEBtFp2bSQubJO9K.jpeg
Zhenmin Weng
user
Tamiflu233
65cadc746938a6a81ff24a9a
Zhoumianze Liu
65cadc746938a6a81ff24a9b
Shunyu Yao
admin_assigned
2024-02-13T08:29:52.228Z
62aa3b8f46d66bafe2f55899
/avatars/d45c6406042c3f6167153aba76d03776.svg
Shunyu Yao
user
ysymyth
65cadc746938a6a81ff24a9c
Tao Yu
65cadc746938a6a81ff24a9d
Lingpeng Kong
OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
Autonomous interaction with the computer has been a longstanding challenge
with great potential, and the recent proliferation of large language models
(LLMs) has markedly accelerated progress in building digital agents. However,
most of these agents are designed to interact with a narrow domain, such as a
specific software or website. This narrow focus constrains their applicability
for general computer tasks. To this end, we introduce OS-Copilot, a framework
to build generalist agents capable of interfacing with comprehensive elements
in an operating system (OS), including the web, code terminals, files,
multimedia, and various third-party applications. We use OS-Copilot to create
FRIDAY, a self-improving embodied agent for automating general computer tasks.
On GAIA, a general AI assistants benchmark, FRIDAY outperforms previous methods
by 35%, showcasing strong generalization to unseen applications via accumulated
skills from previous tasks. We also present numerical and quantitative evidence
that FRIDAY learns to control and self-improve on Excel and Powerpoint with
minimal supervision. Our OS-Copilot framework and empirical findings provide
infrastructure and insights for future research toward more capable and
general-purpose computer agents.
OS-Copilot framework enables FRIDAY, a self-improving generalist agent, to automate a wide range of computer tasks with enhanced generalization and minimal supervision.
LLMs OS-Copilot digital agents embodied agent general AI assistants benchmark GAIA self-improving general computer tasks minimal supervision