Post-mortem of March 12 Google outage
&gt; <a href="https:&#x2F;&#x2F;landing.google.com&#x2F;sre&#x2F;sre-book&#x2F;chapters&#x2F;reliable-product-launches&#x2F;" rel="nofollow">https:&#x2F;&#x2F;landing.google.com&#x2F;sre&#x2F;sre-book&#x2F;chapters&#x2F;reliable-pr...</a><p>&gt; Almost all updates to Google’s services proceed gradually, according to a defined process, with appropriate verification steps interspersed. [..] The first stages of a rollout are usually called &quot;canaries” — an allusion to canaries carried by miners into a coal mine to detect dangerous gases. Our canary servers detect dangerous effects from the behavior of the new software under real user traffic.<p>&gt; Canary testing is a concept embedded into many of Google’s internal tools used to make automated changes, as well as for systems that change configuration files. [..] If the change doesn’t pass the validation period, it’s automatically rolled back.<p>Any Googlers around to explain why Canaries didn&#x27;t catch configuration causing this side effect and stopping the deployment before the issue cascaded globally?<p>The outage here reminds me of issues CloudFront faced wrt configurations in their earlier years. Here&#x27;s master presenter Harvo Jones going into a bit more detail: <a href="https:&#x2F;&#x2F;youtube.com&#x2F;watch?v=n8qQGLJeUYA" rel="nofollow">https:&#x2F;&#x2F;youtube.com&#x2F;watch?v=n8qQGLJeUYA</a><p>Kind of vindicates AWS&#x27; decision (after major outages to DynamoDB and S3) to not just isolate regions or zones but resources within those regions&#x2F;zones further into what they call &quot;cells&quot; <a href="https:&#x2F;&#x2F;youtube.com&#x2F;watch?v=swQbA4zub20" rel="nofollow">https:&#x2F;&#x2F;youtube.com&#x2F;watch?v=swQbA4zub20</a><p>Tumblr spoke abt &quot;cells&quot; too in the context of achieving isolation at scale, but in a quite different way: <a href="http:&#x2F;&#x2F;highscalability.com&#x2F;blog&#x2F;2012&#x2F;2&#x2F;13&#x2F;tumblr-architecture-15-billion-page-views-a-month-and-harder.html" rel="nofollow">http:&#x2F;&#x2F;highscalability.com&#x2F;blog&#x2F;2012&#x2F;2&#x2F;13&#x2F;tumblr-architectur...</a><p>As these known-knowns (and the Google sre-book is teeming with these) get designed around, it&#x27;d be interesting to see why&#x2F;how future outages would happen.