&quot;It unlikely to happen&quot; is not a good strategy to rely on with systems operating at scale. There are about a billion cars on earth traveling trillions of miles every year, many of which will eventually be self-driving. At that scale, you don&#x27;t need a malicious actor working to fool these systems, you just need to encounter the wrong environment. And even if the system is perfect on the day it&#x27;s released, that doesn&#x27;t mean that it will remain so indefinitely (even with proper maintenance).<p>Studying induced failure in neural networks may help us understand the failure modes and mechanisms of these systems.