title: PC Agent: While You Sleep, AI Works -- A Cognitive Journey into Digital
    World
summary: Imagine a world where AI can handle your work while you sleep - organizing
  your research materials, drafting a report, or creating a presentation you need
  for tomorrow. However, while current digital agents can perform simple tasks,
  they are far from capable of handling the complex real-world work that humans
  routinely perform. We present PC Agent, an AI system that demonstrates a
  crucial step toward this vision through human cognition transfer. Our key
  insight is that the path from executing simple "tasks" to handling complex
  "work" lies in efficiently capturing and learning from human cognitive
  processes during computer use. To validate this hypothesis, we introduce three
  key innovations: (1) PC Tracker, a lightweight infrastructure that efficiently
  collects high-quality human-computer interaction trajectories with complete
  cognitive context; (2) a two-stage cognition completion pipeline that
  transforms raw interaction data into rich cognitive trajectories by completing
  action semantics and thought processes; and (3) a multi-agent system combining
  a planning agent for decision-making with a grounding agent for robust visual
  grounding. Our preliminary experiments in PowerPoint presentation creation
  reveal that complex digital work capabilities can be achieved with a small
  amount of high-quality cognitive data - PC Agent, trained on just 133 cognitive
  trajectories, can handle sophisticated work scenarios involving up to 50 steps
  across multiple applications. This demonstrates the data efficiency of our
  approach, highlighting that the key to training capable digital agents lies in
  collecting human cognitive data. By open-sourcing our complete framework,
  including the data collection infrastructure and cognition completion methods,
  we aim to lower the barriers for the research community to develop truly
  capable digital agents.
paper_authors._id: 676a8186b88f71d971444400
paper_authors.name: Yanheng He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:34:38.504Z
paper_authors._id: 661b9ac57cfb7bcb3057a578
paper_authors.avatarUrl: /avatars/f8afaa8eaad3a1e5963a4feebec3f7ab.svg
paper_authors.fullname: Yanheng He
paper_authors.type: user
paper_authors.user: henryhe0123
paper_authors._id: 676a8186b88f71d971444401
paper_authors.name: Jiahe Jin
paper_authors._id: 676a8186b88f71d971444402
paper_authors.name: Shijie Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-06T14:15:31.725Z
paper_authors._id: 65900d4ff5a209eeac08b463
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65900d4ff5a209eeac08b463/PJNNBRJIk1qR24oaRLTex.jpeg
paper_authors.fullname: shijie xia
paper_authors.type: user
paper_authors.user: seven-cat
paper_authors._id: 676a8186b88f71d971444403
paper_authors.name: Jiadi Su
paper_authors._id: 676a8186b88f71d971444404
paper_authors.name: Runze Fan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:34:41.541Z
paper_authors._id: 616bfc2b40e2f69baa1c7add
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/616bfc2b40e2f69baa1c7add/Os7_qgMei-2lRVelrOG7B.jpeg
paper_authors.fullname: Run-Ze Fan
paper_authors.type: user
paper_authors.user: Vfrz
paper_authors._id: 676a8186b88f71d971444405
paper_authors.name: Haoyang Zou
paper_authors._id: 676a8186b88f71d971444406
paper_authors.name: Xiangkun Hu
paper_authors._id: 676a8186b88f71d971444407
paper_authors.name: Pengfei Liu
paper_title: PC Agent: While You Sleep, AI Works -- A Cognitive Journey into Digital
    World
paper_summary: Imagine a world where AI can handle your work while you sleep - organizing
  your research materials, drafting a report, or creating a presentation you need
  for tomorrow. However, while current digital agents can perform simple tasks,
  they are far from capable of handling the complex real-world work that humans
  routinely perform. We present PC Agent, an AI system that demonstrates a
  crucial step toward this vision through human cognition transfer. Our key
  insight is that the path from executing simple "tasks" to handling complex
  "work" lies in efficiently capturing and learning from human cognitive
  processes during computer use. To validate this hypothesis, we introduce three
  key innovations: (1) PC Tracker, a lightweight infrastructure that efficiently
  collects high-quality human-computer interaction trajectories with complete
  cognitive context; (2) a two-stage cognition completion pipeline that
  transforms raw interaction data into rich cognitive trajectories by completing
  action semantics and thought processes; and (3) a multi-agent system combining
  a planning agent for decision-making with a grounding agent for robust visual
  grounding. Our preliminary experiments in PowerPoint presentation creation
  reveal that complex digital work capabilities can be achieved with a small
  amount of high-quality cognitive data - PC Agent, trained on just 133 cognitive
  trajectories, can handle sophisticated work scenarios involving up to 50 steps
  across multiple applications. This demonstrates the data efficiency of our
  approach, highlighting that the key to training capable digital agents lies in
  collecting human cognitive data. By open-sourcing our complete framework,
  including the data collection infrastructure and cognition completion methods,
  we aim to lower the barriers for the research community to develop truly
  capable digital agents.