AWS outage summary
I dunno about you, but I could use a TL;DR for this:1. They fucked up an internal DNS change and didn't notice2. Internal systems on EBS hosts piled up with messages trying to get to the non-existent domain3. Eventually the messages used up all the memory on the EBS hosts, and thousands of EBS hosts began to die simultaneously4. Meanwhile, panicked operators trying to slow down this tidal wave hit the Throttle Everything button5. The throttling was so aggressive the even normal levels of operation became impossible6. The incident was a single AZ, but the throttling was across the whole region, which spread the pain further[Everybody who got throttled gets a 3-hour refund]7. Any single-AZ RDS instance on a dead EBS host was fucked8. Multi-AZ RDS instances ran into two separate bugs, and either became stuck or hit a replication race condition and shut down[Everybody whose multi-AZ RDS didn't fail over gets 10 days free credit]9. Single-AZ ELB instances in the broken AZ failed because they use EBS too10. Because everybody was freaking out and trying to fix their ELBs, the ELB service ran out of IP addresses and locked up11. Multi-AZ ELB instances took too long to notice EBS was broken and then hit a bug and didn't fail over properly anyway[ELB users get no refund, which seems harsh]For those keeping score, that's 1 human error, 2 dependency chains, 3 design flaws, 3 instances of inadequate monitoring, and 5 brand-new internal bugs. From the length and groveling tone of the report, I can only assume that a big chunk of customers are very, VERY angry at them.