Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region
Complex systems are really really hard. I&#x27;m not a big fan of seeing all these folks bash AWS for this, and not really understanding the complexity or nastiness of situations like this. Running the kind of services they do for the kind of customers, this is a VERY hard problem.<p>We ran into a very similar issue, but at the database layer in our company literally 2 weeks ago, where connections to our MySQL exploded and completely took down our data tier and caused a multi-hour outage, compounded by retries and thundering herds. Understanding this problem under the stressful scenario is extremely difficult and a harrowing experience. Anticipating this kind of issue is very very tricky.<p>Naive responses to this include &quot;better testing&quot;, &quot;we should be able to do this&quot;, &quot;why is there no observability&quot; etc. The problem isn&#x27;t testing. Complex systems behave in complex ways, and its difficult to model and predict, especially when the inputs to the system aren&#x27;t entirely under your control. Individual components are easy to understand, but when integrating, things get out of whack. I can&#x27;t stress how difficult it is to model or even think about these systems, they&#x27;re very very hard. Combined with this knowledge being distributed among many people, you&#x27;re dealing with not only distributed systems, but also distributed people, which adds more difficulty in wrapping this around your head.<p>Outrage is the easy response. Empathy and learning is the valuable one. Hugs to the AWS team, and good learnings for everyone.