Automated to Death (2009)
There&#x27;s a weird vibe in this article I don&#x27;t like. It (correctly) notes that as the number of anomalies that the automation passes on to the human operators goes down, the rate that humans successfully handle them also goes down.<p>But it doesn&#x27;t seem to do a good job of clarifying that <i>the total number of incorrectly handled anomalies is still decreasing.</i> Let&#x27;s say your automation goes from handling 90% to 99% of the anomalies and that when it does handle one, it does so correctly. We&#x27;ll say that the increased rarity of human interaction and the inattention and weakened training that causes makes the human pilots to go from being able to handle 90% of them correctly to only a terrifying 40%.<p>Let&#x27;s run the simulation. With the old automation:<p><pre><code>    1000 anomalies occur
     100 (10%) make it past the automation
      10 (10%) make it past the human operators
</code></pre>
So 10 catastrophes. Now with the new moderately better automation and much worse human performance:<p><pre><code>    1000 anomalies occur
      10 (1%) make it past the automation
       6 (60%) make it past the human operators
</code></pre>
6 catastrophes. Even though the human performance was <i>much</i> worse, because they are the last stage in the pipeline, it has a lower effect.<p>Now, I just pulled these numbers out of my ass, but I think it&#x27;s important to focus on the total number of automation+human failures and not single out one stage or the other. From the passenger&#x27;s perspective, they don&#x27;t care who saved their ass, just that it got saved. If we can make one stage more failure proof at the expense of the other, it can still be a net win.