How Organisms Come to Know the World: Fundamental Limits on AGI
I&#x27;ve always thought to develop AGI you really need an evolutionary iterative system that can explore an environment with sensory input and reward feedback for actions. In other words a synthetic analog of a real organism in the real world.<p>Few interesting things come: 1) either try to bootstrap AGI in robots exploring the real world with senses, or try to bootstrap AGI in online robots who have some sort of online senses like scanning text and images, but then you need a set of online &#x27;threats&#x27; to that robot, so that it can have reward feedback--it can&#x27;t just exist &#x27;for free&#x27; it has to &#x27;fight&#x27; to live, otherwise no growth; and 2) you basically need emotions, because emotions are, essentially, our compressed, immediate internal responses to aggregate experiences in the world--they give us a way to interface with our reward feedback, so turns out you need emotions for AGI too, to have an effective interface with reward feedback in a complex environment; and 3) you need a complex environment with plenty of threats in order to drive evolution, and obtaining rewards has to be sufficiently ill-defined, difficult but possible that one has to figure out some unique solution for oneself; the world is just as important as the &#x27;brain&#x27; for AGI to develop. If you (horribly) took a new born baby and kept it in a white featureless room (or in some sort of contrived abstract environment), and met its survival needs , it would grow up skewed and probably not very &#x27;generally intelligent.&#x27;<p>AGI doesn&#x27;t have to be the same as consciousness...but you need something that &#x27;lives in a world&#x27; and has limitations that it has to figure out how to overcome, and emotional responses in order to represent and respond to experiences, in order to develop it.<p>When you realize this, it&#x27;s kind of funny to think of the trope that AGI is always represented like this &quot;robotic emotionless logical&quot;, but I think that&#x27;s impossible. That&#x27;s more like nth-gen ChatGPT, than real AGI. I think real AGI needs to be emotional--maybe not the same as human emotions, but it has to have them, and has to have a world to grow in.