The tech world is rallying around a young developer who made a huge mistake
Unrelated to the story, but I&#x27;ve had a similar experience as to some of the stories described in the article - where a cascade of unseen problems landed on my lap and I took the blame even though little of it was my doing.<p>(Note, this was many years ago and mildly traumatic, so some details might be fuzzy)<p>In 2001 I was working for one of the larger internet server co-location companies in the US as a dedicated customer support engineer (remote hands) for several large clients.<p>One of my clients requested that I perform a memory upgrade on their Sun 420R servers at 4am (on my birthday none the less).<p>So I come in.   Confirm with the client they are ready to be shut down, confirm with the NOC that the servers are coming down.    The client remotely shut down the servers in question but they didn&#x27;t power off (missed clue).   One by one I power them down, roll them out of the rack, insert memory.<p>This was a pretty routine request, so it went smoothly - so I had thought.<p>Switch the console to the database server, boot.  The kernel starts without issue, then a slew of error messages.  Too many to read.<p>The Sun 420R&#x27;s were known to have memory riser issues, so I power off reseat the riser, and all the memory.   Same problem.<p>I take out the new memory.  Same problem.<p>Before I could troubleshoot the client guesses it&#x27;s a hardware issue, suggests we bring up the other systems and the cold-backup systems to compensate.<p>Nothing comes up.<p>Not<p>One<p>System<p>So at this point the client starts to panic over the phone.  They hang up on me.<p>So I pull out my laptop and switch to a serial console on one of the servers in attempt to troubleshoot the error messages.<p>All the error messages are Bourne Shell (sh, not bash) errors.<p>Realizing this doesn&#x27;t appear to be a hardware issue, I boot from a recovery cdrom, and start inspecting the init system.  Everything looked great.   So I tried to run a script, and then it became obvious - everything LOOKED too great.<p>All the init files had been reformatted.   It said as much in a comment.   A certain very well known perl developer (at the time) who developed the site for the client had found the formatting of the init scripts atrocious and unreadable, so took it upon himself to reformat every init script on every server.   The problem is, bourne shell didn&#x27;t like some of the formatting changes - mainly space between variable, equals sign, and value declarations.   Had he tested his changes, it would have been obvious what the problem was.<p>Testing my theory I fixed the formatting of a single script, and it immediately works without error.  Problem found, now I need to fix all the scripts on all servers.<p>I call the client.   No answer.<p>I call the client again.  Answer.   &quot;Did you fix our servers yet?&quot;   &quot;No, I found...&quot;  phone disconnects.<p>I call again, no answer.<p>I call again, no answer.<p>I call the NOC, asking they try to inform the client of the status.<p>So I proceed to start fixing the first server.   It took an hour, but the server came up.   The second server came up a little faster, and I proceeded to work on the third and fourth.<p>Meanwhile, I find out the client had been calling around escalating to senior management at my employer petitioning that they fire me and put someone competent on the job.<p>I find this out as one of the analysts comes up from the NOC and tells me I&#x27;m relieved, and was asked to go home with no further information.   So I explain what had happened and what I was doing to fix the issue.<p>The next day I am called into the office.  Told I was fired for my obvious screwups, and that thankfully the NOC analyst had been there to save them from losing such a large client.<p>Happy birthday.<p>Without question, there were things I could have done better in that situation, but could not have predicted in a million years that someone would reformat init files without testing them.