This is a really great question since the vast majority of the work was in ensuring correctness and reliability: everything from testing discipline to fault injection to auditing. This also included hardware testing, like pulling out circuit breakers to test our power distribution, or overheating a rack to test graceful shutdown.<p>I&#x27;ll give a slightly lazy answer here however and point you to a talk I gave about building durable systems, which covers a lot of this material:
<a href="https:&#x2F;&#x2F;www.oreilly.com&#x2F;events&#x2F;velocity&#x2F;devops-web-performance-ny-2015&#x2F;sessions&#x2F;46120&#x2F;durability-theater-or-real-world-safety-how-to-engineer-against-data-loss" rel="nofollow">https:&#x2F;&#x2F;www.oreilly.com&#x2F;events&#x2F;velocity&#x2F;devops-web-performan...</a>