The problem is that unlike with &quot;regular&quot; engineering, there are mathematical theorems that basically say that writing bug-free code is somewhere between prohibitively expensive and impossible. The cost of proving a non-trivial program property has a proven lower bound of O(S) where S is the number of states the program may take; alternatively, we say that proving correctness is <i>at least</i> PSPACE-complete (if not EXPTIME) in the size of the program. Not saying we can&#x27;t do our best and use various tools, but this cost cannot -- in general -- be reduced to anything acceptable, and therefore we will never be shocked when a bug does occur.<p>It is easier to achieve string AI than to write bug-free code, and even that won&#x27;t help: a super-intelligence will still have bugs in the code it writes.