Discord outage postmortem
&gt; Our service discovery system is resilient to various failure modes within etcd, however, we did not anticipate a client being able to write a corrupt record due to improper HTTP handling. golang&#x27;s net&#x2F;http module — which is in violation of the HTTP 1.1 specification in this specific case — introduced an additional failure case which we did not anticipate. It is expected that a Content-Length header and an incomplete body should be rejected by a server as being incomplete. Unfortunately net&#x2F;http does not check that the bytes read from the body portion of the request match the Content-Length header.<p>Whichever engineer wrote puts too much weight on the theoretical aspects of HTTP.<p>The reality is that any of Discord&#x27;s software could have had erroneous code that announced an empty service list. This would have crashed all of Discord regardless -- transient network errors and mismatched HTTP Content-Length headers <i>need not apply.</i><p>A <i>resilient</i> HTTP server will try to handle whatever the HTTP client sends it, and then the specific endpoint can enforce whatever levels of strictness are needed by the application. It&#x27;s nice to be able to point a finger at &quot;not my code&quot;, but how many popular web server libraries (not standalone web servers like nginx or apache) out there will actually flat out reject the entire request because of a simple Content-Length mismatch, rather than handing that decision off to the endpoint?<p>If an empty string is not a valid value for your service discovery system, you should reject those from the database with a CHECK CONSTRAINT if possible, or with the code that sits in front of the database if CONSTRAINTS don&#x27;t exist (like I believe is the case for etcd). It&#x27;s also not a bad idea for the clients to validate what they receive and filter out values they can&#x27;t handle to enhance robustness.<p>You definitely shouldn&#x27;t rely on nothing ever <i>attempting</i> to put a bad value into the database.<p>The web server&#x27;s willingness to accept &quot;strictly&quot; malformed requests when the engineers were unaware of this fact was certainly a contributing factor, but I don&#x27;t think it was the root cause in this case, as they&#x27;re heavily trying to suggest.<p>(I think it would be neat to have a &quot;strict&quot; mode that you could turn on for every web server, especially when being used as an internally-facing service, but most companies want to make it as easy as possible for their customers&#x2F;partners to integrate with their service... and that means accepting requests that aren&#x27;t always academically perfect.)