How to lose $172k per second for 45 minutes (2013)
discussed previously at:<p><a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=6589508" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=6589508</a><p>I remember the week after this.  Everyone I knew who worked at a fund was going over their code and also updating their Compliance documents covering testing and deployment of automated code.<p><i></i>As a side note<i></i> one of hte biggest ways funds tend to get in trouble from their regulators is to not follow the steps outlined in their compliance manual.  Its been my experience that regulators care more that you follow the steps in your manual than those steps necessary being the best way to do something.<p>I came away from this thinking the worst part of this was that their system did send them errors, its just that when you deal with billions of events emailing errors just tend to get ignored as at that scale you generate so many false positives with logging.<p>I still don&#x27;t know the best way to  monitor and alert users for large distributed systems.<p>The other take away was that this wasn&#x27;t just a software issue but a deployment issue as well.  It wasn&#x27;t just one root cause but a number of issues that built up to cause the issue.<p>1) New exchange feature going live so this is the first day you are actually running live with this feature<p>2) old code left in the system long after it was done being used<p>3) re-purposed command flag that used to call the old code, but now is used in the new code<p>4) only a partial deployment leaving both old and new code working together.<p>5) inability to quickly diagnose where the problem was<p>6) you are also  managing client orders and have the equivalent of an SLA with them so you don&#x27;t want to go nuclear and shut down everything