title: Evolution and The Knightian Blindspot of Machine Learning
summary: This paper claims that machine learning (ML) largely overlooks an important
  facet of general intelligence: robustness to a qualitatively unknown future in
  an open world. Such robustness relates to Knightian uncertainty (KU) in
  economics, i.e. uncertainty that cannot be quantified, which is excluded from
  consideration in ML's key formalisms. This paper aims to identify this blind
  spot, argue its importance, and catalyze research into addressing it, which we
  believe is necessary to create truly robust open-world AI. To help illuminate
  the blind spot, we contrast one area of ML, reinforcement learning (RL), with
  the process of biological evolution. Despite staggering ongoing progress, RL
  still struggles in open-world situations, often failing under unforeseen
  situations. For example, the idea of zero-shot transferring a self-driving car
  policy trained only in the US to the UK currently seems exceedingly ambitious.
  In dramatic contrast, biological evolution routinely produces agents that
  thrive within an open world, sometimes even to situations that are remarkably
  out-of-distribution (e.g. invasive species; or humans, who do undertake such
  zero-shot international driving). Interestingly, evolution achieves such
  robustness without explicit theory, formalisms, or mathematical gradients. We
  explore the assumptions underlying RL's typical formalisms, showing how they
  limit RL's engagement with the unknown unknowns characteristic of an
  ever-changing complex world. Further, we identify mechanisms through which
  evolutionary processes foster robustness to novel and unpredictable challenges,
  and discuss potential pathways to algorithmically embody them. The conclusion
  is that the intriguing remaining fragility of ML may result from blind spots in
  its formalisms, and that significant gains may result from direct confrontation
  with the challenge of KU.
paper_authors._id: 6791ae54330198cc26b72479
paper_authors.name: Joel Lehman
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-01-23T02:49:57.110Z
paper_authors._id: 6444241e9c1bd83bd19ea70f
paper_authors.avatarUrl: /avatars/24b4e65f26f5f8dcc1465cef67fd334b.svg
paper_authors.fullname: Joel Lehman
paper_authors.type: user
paper_authors.user: jal278
paper_authors._id: 6791ae54330198cc26b7247a
paper_authors.name: Elliot Meyerson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-24T10:20:51.128Z
paper_authors._id: 6514b7fde1273c28705142cc
paper_authors.avatarUrl: /avatars/072bf14abd8ef17d9393338a20157cc2.svg
paper_authors.fullname: Elliot Meyerson
paper_authors.type: user
paper_authors.user: ekmeyerson
paper_authors._id: 6791ae54330198cc26b7247b
paper_authors.name: Tarek El-Gaaly
paper_authors._id: 6791ae54330198cc26b7247c
paper_authors.name: Kenneth O. Stanley
paper_authors._id: 6791ae54330198cc26b7247d
paper_authors.name: Tarin Ziyaee
paper_title: Evolution and The Knightian Blindspot of Machine Learning
paper_summary: This paper claims that machine learning (ML) largely overlooks an important
  facet of general intelligence: robustness to a qualitatively unknown future in
  an open world. Such robustness relates to Knightian uncertainty (KU) in
  economics, i.e. uncertainty that cannot be quantified, which is excluded from
  consideration in ML's key formalisms. This paper aims to identify this blind
  spot, argue its importance, and catalyze research into addressing it, which we
  believe is necessary to create truly robust open-world AI. To help illuminate
  the blind spot, we contrast one area of ML, reinforcement learning (RL), with
  the process of biological evolution. Despite staggering ongoing progress, RL
  still struggles in open-world situations, often failing under unforeseen
  situations. For example, the idea of zero-shot transferring a self-driving car
  policy trained only in the US to the UK currently seems exceedingly ambitious.
  In dramatic contrast, biological evolution routinely produces agents that
  thrive within an open world, sometimes even to situations that are remarkably
  out-of-distribution (e.g. invasive species; or humans, who do undertake such
  zero-shot international driving). Interestingly, evolution achieves such
  robustness without explicit theory, formalisms, or mathematical gradients. We
  explore the assumptions underlying RL's typical formalisms, showing how they
  limit RL's engagement with the unknown unknowns characteristic of an
  ever-changing complex world. Further, we identify mechanisms through which
  evolutionary processes foster robustness to novel and unpredictable challenges,
  and discuss potential pathways to algorithmically embody them. The conclusion
  is that the intriguing remaining fragility of ML may result from blind spots in
  its formalisms, and that significant gains may result from direct confrontation
  with the challenge of KU.