Superintelligence Cannot Be Contained: Lessons from Computability Theory
I don&#x27;t quite accept the argument. You can demonstrate a system to be isolated. For example, if you put a computer in a metallic box with a battery inside, you can compute the maximum electric field generated inside, and choosing the appropriate radius you can guarantee no significant influence would be had outside. Of course, you can&#x27;t guarantee no one will breach your containment from the outside, essentially freeing whatever intelligence to influence the world. But this goes beyond the danger of the intelligence itself, more the general infeasibility of predicting the future without having much greater computational power than the entire world.<p>Likewise, the alignment problem isn&#x27;t necessarily unsolvable. Since we can design simple systems that follow expectations exactly, why couldn&#x27;t we design complex systems that also follow expectations? Of course, the design of expectations itself gets fuzzy, making this a failure point, but to the extent that we can define good and evil I see no impediment to align an intelligence to them.<p>(In fact, I believe &#x27;control&#x27; or &#x27;containment&#x27; is a severely misguided proposition. We want to really assure the core motivation of such machines similar to ours; not create conflicting motivations and secure it via containment. This is indeed extremely dangerous, and potentially unethical: if machines approach consciousness, wouldn&#x27;t such &#x27;containment&#x27; be similar to slavery?)