Superintelligence: The Idea That Eats Smart People
So many people don't realize that as social creatures, we see the world through the lens of our social brains. Computers don't care about "all powerful" or "savant" or "should". They don't qualitatively distinguish between killing one person and every person. Self-replication plus proliferation of cheap components plus proliferation of AI algorithms equals a time when a script kiddie or a bug can mean every last fragile sack of meat and water gets punctured or irradiated or whatever. If it can be done in a video game with good physics simulation, it can be done eventually in real life. It won't be like a movie where the ticking time bomb works on a human timescale and always has a humanlike weakness. Comparing this to nuclear weapons is silly. It's more like issuing every person in the world a "kill or help between 0 and 7 billion people" button that's glitchy and spinning up 1,000 4chan chains with advice on tinkering with it.