title: Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer
    a Safer Path?
summary: The leading AI companies are increasingly focused on building generalist AI
  agents -- systems that can autonomously plan, act, and pursue goals across
  almost all tasks that humans can perform. Despite how useful these systems
  might be, unchecked AI agency poses significant risks to public safety and
  security, ranging from misuse by malicious actors to a potentially irreversible
  loss of human control. We discuss how these risks arise from current AI
  training methods. Indeed, various scenarios and experiments have demonstrated
  the possibility of AI agents engaging in deception or pursuing goals that were
  not specified by human operators and that conflict with human interests, such
  as self-preservation. Following the precautionary principle, we see a strong
  need for safer, yet still useful, alternatives to the current agency-driven
  trajectory. Accordingly, we propose as a core building block for further
  advances the development of a non-agentic AI system that is trustworthy and
  safe by design, which we call Scientist AI. This system is designed to explain
  the world from observations, as opposed to taking actions in it to imitate or
  please humans. It comprises a world model that generates theories to explain
  data and a question-answering inference machine. Both components operate with
  an explicit notion of uncertainty to mitigate the risks of overconfident
  predictions. In light of these considerations, a Scientist AI could be used to
  assist human researchers in accelerating scientific progress, including in AI
  safety. In particular, our system can be employed as a guardrail against AI
  agents that might be created despite the risks involved. Ultimately, focusing
  on non-agentic AI may enable the benefits of AI innovation while avoiding the
  risks associated with the current trajectory. We hope these arguments will
  motivate researchers, developers, and policymakers to favor this safer path.
paper_authors._id: 67bbfd6c3593f69f41512d54
paper_authors.name: Yoshua Bengio
paper_authors._id: 67bbfd6c3593f69f41512d55
paper_authors.name: Michael Cohen
paper_authors._id: 67bbfd6c3593f69f41512d56
paper_authors.name: Damiano Fornasiere
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:53:43.499Z
paper_authors._id: 63e4beb2d6278c161be4ef52
paper_authors.avatarUrl: /avatars/15bbfde42e890f6f0dd0efd32dfdf5fa.svg
paper_authors.fullname: Damiano Fornasiere
paper_authors.type: user
paper_authors.user: dfp00
paper_authors._id: 67bbfd6c3593f69f41512d57
paper_authors.name: Joumana Ghosn
paper_authors._id: 67bbfd6c3593f69f41512d58
paper_authors.name: Pietro Greiner
paper_authors._id: 67bbfd6c3593f69f41512d59
paper_authors.name: Matt MacDermott
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:53:56.982Z
paper_authors._id: 65bb7386a951088f7dc69b16
paper_authors.avatarUrl: /avatars/bb72822451a9d769951c03e0cc3b1912.svg
paper_authors.fullname: Matt MacDermott
paper_authors.type: user
paper_authors.user: mattmacdermott
paper_authors._id: 67bbfd6c3593f69f41512d5a
paper_authors.name: Sören Mindermann
paper_authors._id: 67bbfd6c3593f69f41512d5b
paper_authors.name: Adam Oberman
paper_authors._id: 67bbfd6c3593f69f41512d5c
paper_authors.name: Jesse Richardson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:54:09.707Z
paper_authors._id: 6602268ec92df8c189d56ef1
paper_authors.avatarUrl: /avatars/ec4a88bbf01f226640725ac117e53eae.svg
paper_authors.fullname: Jesse Richardson
paper_authors.type: user
paper_authors.user: getfull
paper_authors._id: 67bbfd6c3593f69f41512d5d
paper_authors.name: Oliver Richardson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:54:15.449Z
paper_authors._id: 64e3e41e52a2eece10b99f0f
paper_authors.avatarUrl: /avatars/90045fe790388fe2cd010e04ad0137d1.svg
paper_authors.fullname: Oliver Richardson
paper_authors.type: user
paper_authors.user: olliekse
paper_authors._id: 67bbfd6c3593f69f41512d5e
paper_authors.name: Marc-Antoine Rondeau
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:54:22.066Z
paper_authors._id: 67112a441635565bd8d0a6cd
paper_authors.avatarUrl: /avatars/68dc8aa2a65c47c2e0869119212ac4aa.svg
paper_authors.fullname: Marc-Antoine Rondeau
paper_authors.type: user
paper_authors.user: marondeau
paper_authors._id: 67bbfd6c3593f69f41512d5f
paper_authors.name: Pierre-Luc St-Charles
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:54:28.067Z
paper_authors._id: 6686a17d7c05938b94d81875
paper_authors.avatarUrl: /avatars/caff2bc79068a024d74e9d3b7ea79eaf.svg
paper_authors.fullname: Pierre-Luc St-Charles
paper_authors.type: user
paper_authors.user: plstcharles-mila
paper_authors._id: 67bbfd6c3593f69f41512d60
paper_authors.name: David Williams-King
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T15:54:34.058Z
paper_authors._id: 6776217bd8ace1de4692e17e
paper_authors.avatarUrl: /avatars/357eb3e2c96b8a98b5d1029eefbb0ed3.svg
paper_authors.fullname: David Williams-King
paper_authors.type: user
paper_authors.user: dwksrc
paper_title: Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer
    a Safer Path?
paper_summary: The leading AI companies are increasingly focused on building generalist AI
  agents -- systems that can autonomously plan, act, and pursue goals across
  almost all tasks that humans can perform. Despite how useful these systems
  might be, unchecked AI agency poses significant risks to public safety and
  security, ranging from misuse by malicious actors to a potentially irreversible
  loss of human control. We discuss how these risks arise from current AI
  training methods. Indeed, various scenarios and experiments have demonstrated
  the possibility of AI agents engaging in deception or pursuing goals that were
  not specified by human operators and that conflict with human interests, such
  as self-preservation. Following the precautionary principle, we see a strong
  need for safer, yet still useful, alternatives to the current agency-driven
  trajectory. Accordingly, we propose as a core building block for further
  advances the development of a non-agentic AI system that is trustworthy and
  safe by design, which we call Scientist AI. This system is designed to explain
  the world from observations, as opposed to taking actions in it to imitate or
  please humans. It comprises a world model that generates theories to explain
  data and a question-answering inference machine. Both components operate with
  an explicit notion of uncertainty to mitigate the risks of overconfident
  predictions. In light of these considerations, a Scientist AI could be used to
  assist human researchers in accelerating scientific progress, including in AI
  safety. In particular, our system can be employed as a guardrail against AI
  agents that might be created despite the risks involved. Ultimately, focusing
  on non-agentic AI may enable the benefits of AI innovation while avoiding the
  risks associated with the current trajectory. We hope these arguments will
  motivate researchers, developers, and policymakers to favor this safer path.
paper_ai_summary: A proposal advocates for developing a non-agentic AI system, called Scientist AI, to enhance safety and trustworthiness while advancing scientific research and mitigating risks from autonomous AI agents.
paper_ai_keywords: AI agents generalist AI AI agency public safety security misuse irreversible loss AI training methods deception goal pursuit human interests self-preservation precautionary principle non-agentic AI trustworthy by design world model theories data question-answering inference machine uncertainty AI safety guardrail AI innovation