Ask HN: How do I feel less guilty about bugs?
I&#x27;m responding _before_ reading the comments, because this is something I feel like I have some fairly strong opinions on.<p>I see two discrete issues here. I&#x27;ll post one here, and one in another comment because I&#x27;m apparently long-winded today :).<p>First, as a developer, you _will_ break things. There is absolutely no getting around that - it will happen. You can&#x27;t choose not to break things. You _can_ choose to learn from it.<p>I&#x27;ve broken production systems <i>many</i> times, and will break them in the future. I&#x27;ve been writing software for sixteen years now; I break things less often now than I did when I was more junior, but I would say &quot;frequency&quot; is the biggest difference that comes with experience. Instead, I tend not to break things in the same ways.<p>When I review my work (and the work of teammates) I do so with the critical eye of someone who&#x27;s seen how things are likely to break. I also look for ways that things are likely to go wrong and pre-plan - to the extent that&#x27;s possible - how they can be fixed.<p>Finally, I do everything I can to make it easy to troubleshoot and recover when things (invariably) go wrong. I have a very thorough and &quot;intentional&quot; plan for when things are on fire, and always pause before taking a remediation step.<p>For example, I once deleted a major project on a scientific publishing platform. The company was very new, and while we had backups, the recovery plan was not exercised and significant work would have been lost if I&#x27;d used it. In this case, I was intending to update a single nested attribute on a MongoDB object, but used the wrong operator and replaced the entire document with the single attribute I wanted to update. As soon as I saw what happened, I took a deep breath, and made sure my next action was <i>intentional</i>. My first instinct was to call over the founder and tell them what happened. I estimated that it would take about 5-10 minutes to explain the issue, so I considered my other options. Restoring from backup would take too long, but mounting a backup on another DB was something I could trigger in a few minutes. I sat that aside for the moment as well, but remembered that I should initiate that if I thought I might need it later. Then I looked at what I had in my shell. I scrolled back and found that I had read the document in the Mongo shell a few minutes before performing the update... and the entire document was right there in the output! I took a picture of the screen with my phone. Yes, that&#x27;s silly, but I wanted to make sure that I didn&#x27;t misclick or something and completely lose the output. Then I took a screenshot on my MBP, and then I copied the output from the terminal and pasted it into a new text document in a new terminal. I saved that, then opened it in a third shell and verified that it was in fact there. Back to another shell, I built the command I would use to restore the object in Vim. It looked good. I connected to my local database, pasted the command, and executed. It worked! I verified that the data looked like it should, and it did. I then pasted the command into production, visually verified that it was correct, and executed. It worked!<p>In the above, the Total time from &quot;Oh crap!&quot; to &quot;Whew!&quot; was less than five minutes. As I mentioned, it would have taken me at least that long to call over a more senior person and bring them up to speed. I stood by my decision not to bring others into the emergency immediately for that reason, but that doesn&#x27;t mean that I was &quot;hiding&quot; anything. Once I saw that it had been fixed, I sent them a Slack message to the effect of &quot;I just broke Project X by overwriting the document in MongoDB. I believe I have resolved it, but please verify it thoroughly ASAP. I&#x27;m writing a summary of what happened right now.&quot; I then wrote the summary and sent it to him. That summary ended up being the basis for a presentation that I gave at the following all-hands meeting.