Evolution and The Knightian Blindspot of Machine Learning
This paper claims that machine learning (ML) largely overlooks an important
facet of general intelligence: robustness to a qualitatively unknown future in
an open world. Such robustness relates to Knightian uncertainty (KU) in
economics, i.e. uncertainty that cannot be quantified, which is excluded from
consideration in ML's key formalisms. This paper aims to identify this blind
spot, argue its importance, and catalyze research into addressing it, which we
believe is necessary to create truly robust open-world AI. To help illuminate
the blind spot, we contrast one area of ML, reinforcement learning (RL), with
the process of biological evolution. Despite staggering ongoing progress, RL
still struggles in open-world situations, often failing under unforeseen
situations. For example, the idea of zero-shot transferring a self-driving car
policy trained only in the US to the UK currently seems exceedingly ambitious.
In dramatic contrast, biological evolution routinely produces agents that
thrive within an open world, sometimes even to situations that are remarkably
out-of-distribution (e.g. invasive species; or humans, who do undertake such
zero-shot international driving). Interestingly, evolution achieves such
robustness without explicit theory, formalisms, or mathematical gradients. We
explore the assumptions underlying RL's typical formalisms, showing how they
limit RL's engagement with the unknown unknowns characteristic of an
ever-changing complex world. Further, we identify mechanisms through which
evolutionary processes foster robustness to novel and unpredictable challenges,
and discuss potential pathways to algorithmically embody them. The conclusion
is that the intriguing remaining fragility of ML may result from blind spots in
its formalisms, and that significant gains may result from direct confrontation
with the challenge of KU.
6791ae54330198cc26b72479
Joel Lehman
extracted_pending
2025-01-23T02:49:57.110Z
6444241e9c1bd83bd19ea70f
/avatars/24b4e65f26f5f8dcc1465cef67fd334b.svg
Joel Lehman
user
jal278
6791ae54330198cc26b7247a
Elliot Meyerson
admin_assigned
2025-01-24T10:20:51.128Z
6514b7fde1273c28705142cc
/avatars/072bf14abd8ef17d9393338a20157cc2.svg
Elliot Meyerson
user
ekmeyerson
6791ae54330198cc26b7247b
Tarek El-Gaaly
6791ae54330198cc26b7247c
Kenneth O. Stanley
6791ae54330198cc26b7247d
Tarin Ziyaee
Evolution and The Knightian Blindspot of Machine Learning
This paper claims that machine learning (ML) largely overlooks an important
facet of general intelligence: robustness to a qualitatively unknown future in
an open world. Such robustness relates to Knightian uncertainty (KU) in
economics, i.e. uncertainty that cannot be quantified, which is excluded from
consideration in ML's key formalisms. This paper aims to identify this blind
spot, argue its importance, and catalyze research into addressing it, which we
believe is necessary to create truly robust open-world AI. To help illuminate
the blind spot, we contrast one area of ML, reinforcement learning (RL), with
the process of biological evolution. Despite staggering ongoing progress, RL
still struggles in open-world situations, often failing under unforeseen
situations. For example, the idea of zero-shot transferring a self-driving car
policy trained only in the US to the UK currently seems exceedingly ambitious.
In dramatic contrast, biological evolution routinely produces agents that
thrive within an open world, sometimes even to situations that are remarkably
out-of-distribution (e.g. invasive species; or humans, who do undertake such
zero-shot international driving). Interestingly, evolution achieves such
robustness without explicit theory, formalisms, or mathematical gradients. We
explore the assumptions underlying RL's typical formalisms, showing how they
limit RL's engagement with the unknown unknowns characteristic of an
ever-changing complex world. Further, we identify mechanisms through which
evolutionary processes foster robustness to novel and unpredictable challenges,
and discuss potential pathways to algorithmically embody them. The conclusion
is that the intriguing remaining fragility of ML may result from blind spots in
its formalisms, and that significant gains may result from direct confrontation
with the challenge of KU.
ML, particularly RL, lacks robustness in open-world scenarios due to its exclusion of Knightian uncertainty, and this paper argues that incorporating evolutionary mechanisms could enhance ML's ability to handle unpredictable challenges.
reinforcement learning Knightian uncertainty open-world AI zero-shot transfer out-of-distribution evolutionary processes