Ask HN: Best “I brought down production” story?
Back in the days of MyISAM and before Google had their own ad network I worked for the world's largest advertising network.  It had a global reach of 75%, meaning 3 / 4s of people saw at least one of our ads daily.<p>I was trying to learn MySQL and the CTO made the mistake of giving me access to the prod database.  This huge network that served most of the ads in the world ran off of only two huge servers running in an office outside Los Angeles.<p>MyISAM uses a read lock on every SELECT query.  I did not know this at the time.  I was running a number of queries that were trying to pull historical performance data for all our ads across all time.  They were taking a long time so I let them run in the background while working on a spreadsheet somewhere else.<p>A little while later I hear some murmuring.  Apparently the whole network was down.  The engineering team was frantically trying to find the cause of the problem.  Eventually, the CTO approaches my desk.  "Were you running some queries on the database?"  "Yes."  "The query you ran was trying to generate billions of rows of results and locked up the entire database.  Roughly three quarters of the ads in the world have been gone for almost two hours."<p>After the second time I did this, he showed me the MySQL EXPLAIN command and I finally twigged that some kinds of JOINs can go exponential.<p>Kudos to him for never revoking my access and letting me learn things the hard way.  Also, if he worked for me I would have fired him.