Post-mortem of March 12 Google outage
> <a href="https://landing.google.com/sre/sre-book/chapters/reliable-product-launches/" rel="nofollow">https://landing.google.com/sre/sre-book/chapters/reliable-pr...</a><p>> Almost all updates to Google’s services proceed gradually, according to a defined process, with appropriate verification steps interspersed. [..] The first stages of a rollout are usually called "canaries” — an allusion to canaries carried by miners into a coal mine to detect dangerous gases. Our canary servers detect dangerous effects from the behavior of the new software under real user traffic.<p>> Canary testing is a concept embedded into many of Google’s internal tools used to make automated changes, as well as for systems that change configuration files. [..] If the change doesn’t pass the validation period, it’s automatically rolled back.<p>Any Googlers around to explain why Canaries didn't catch configuration causing this side effect and stopping the deployment before the issue cascaded globally?<p>The outage here reminds me of issues CloudFront faced wrt configurations in their earlier years. Here's master presenter Harvo Jones going into a bit more detail: <a href="https://youtube.com/watch?v=n8qQGLJeUYA" rel="nofollow">https://youtube.com/watch?v=n8qQGLJeUYA</a><p>Kind of vindicates AWS' decision (after major outages to DynamoDB and S3) to not just isolate regions or zones but resources within those regions/zones further into what they call "cells" <a href="https://youtube.com/watch?v=swQbA4zub20" rel="nofollow">https://youtube.com/watch?v=swQbA4zub20</a><p>Tumblr spoke abt "cells" too in the context of achieving isolation at scale, but in a quite different way: <a href="http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html" rel="nofollow">http://highscalability.com/blog/2012/2/13/tumblr-architectur...</a><p>As these known-knowns (and the Google sre-book is teeming with these) get designed around, it'd be interesting to see why/how future outages would happen.