Pausing AI Developments Isn't Enough. We Need to Shut It All Down
While I think we should take AI risk seriously, I am skeptical of strong claims like the following:<p><i>&gt;the most likely result of building a superhumanly smart AI, under anything remotely like the current circumstances, is that literally everyone on Earth will die</i><p>I&#x27;ve never heard a justification for this claim that isn&#x27;t extremely vague and hand-wavy. Since we have no idea today how AGI would work, how can we assign probabilities to different scenarios involving vastly different hardware requirements, scaling characteristics, and unforeseen scientific breakthroughs?<p>Just a few examples. What is the probability of a take-off that takes weeks vs. a take-off that takes decades? What is the probability that current ML approaches are a dead end and one of the necessary breakthroughs to build AGI will make the alignment problem trivial? What is the probability that AGI will be benevolent to humans? If AGI is in fact a catastrophic risk, what is the probability that it only kills 10%&#x2F;50%&#x2F;99% of humans rather than literally everyone on Earth?<p>I am agnostic about these questions. In my opinion, if you have a high degree of confidence that you have the right answers, there is something wrong with your epistemology.