Post Mortem of Google Outage on 14 December 2020
I've been bitten by quota grace periods before, they're the "buffer bloat" of cloud platform management systems. You build something, it seems fine, and then a month later it keels over.My tiny disaster was caused by Amazon EFS. They provide a performance quota grace period of a month, during which all tests passed with flying colours. Turns out that EFS rations out IOPS per GB of stored data, and this particular application stored only a few MB at the time, because it was brand new and hadn't accumulated anything yet. I had a very angry customer calling up asking me to please explain why they were seeing an average of 0.1 IOPS...From: https://docs.aws.amazon.com/efs/latest/ug/performance.html"The baseline rate is 50 MiB/s per TiB of storage (equivalently, 50 KiB/s per GiB of storage).AWS now provides a performance floor of 1 MiB/s, but at the time there was no floor. If I remember correctly, this application had something like 2 MIB of data, which was constantly being updated by various processes, so there was no quota being accumulated. The system performance went from something like 1 Gbps to 100 bytes per second instantly. It took 10 seconds for a 1 KiB I/O to complete. Fun times, fun times...