(Tedious disclaimer: my opinion, not speaking for my employer, etc)<p>I&#x27;m an SRE at Google, where postmortems are habitual. The thing that jumped out at me here is that a production change was instantaneously pushed globally, instead of being canaried on a fraction of the serving capacity so that problems could be detected. That seems like your big problem here.<p>(Of course, without knowing how your data storage works, it&#x27;s difficult to tell how hard it is to fix that.)