Unix Admin Horror Story Summary (1992)
I found my favorite story buried in the middle of this, from 1986. It&#x27;s a classic on par with The Story of Mel, a Real Programmer. Reproduced here for your reading pleasure:<p><pre><code>  Have you ever left your terminal logged in, only to find when you came
  back to it that a (supposed) friend had typed &quot;rm -rf ~&#x2F;*&quot; and was
  hovering over the keyboard with threats along the lines of &quot;lend me a
  fiver &#x27;til Thursday, or I hit return&quot;?  Undoubtedly the person in
  question would not have had the nerve to inflict such a trauma upon
  you, and was doing it in jest.  So you&#x27;ve probably never experienced the
  worst of such disasters....
  
  It was a quiet Wednesday afternoon.  Wednesday, 1st October, 15:15
  BST, to be precise, when Peter, an office-mate of mine, leaned away
  from his terminal and said to me, &quot;Mario, I&#x27;m having a little trouble
  sending mail.&quot;  Knowing that msg was capable of confusing even the
  most capable of people, I sauntered over to his terminal to see what
  was wrong.  A strange error message of the form (I forget the exact
  details) &quot;cannot access &#x2F;foo&#x2F;bar for userid 147&quot; had been issued by
  msg.  My first thought was &quot;Who&#x27;s userid 147?; the sender of the
  message, the destination, or what?&quot;  So I leant over to another
  terminal, already logged in, and typed
          grep 147 &#x2F;etc&#x2F;passwd
  only to receive the response
          &#x2F;etc&#x2F;passwd: No such file or directory.
  
  Instantly, I guessed that something was amiss.  This was confirmed
  when in response to
          ls &#x2F;etc
  I got
          ls: not found.
  
  I suggested to Peter that it would be a good idea not to try anything
  for a while, and went off to find our system manager.
  
  When I arrived at his office, his door was ajar, and within ten
  seconds I realised what the problem was.  James, our manager, was
  sat down, head in hands, hands between knees, as one whose world has
  just come to an end.  Our newly-appointed system programmer, Neil, was
  beside him, gazing listlessly at the screen of his terminal.  And at
  the top of the screen I spied the following lines:
          # cd
          # rm -rf *
  
  Oh, shit, I thought.  That would just about explain it.
  
  I can&#x27;t remember what happened in the succeeding minutes; my memory is
  just a blur.  I do remember trying ls (again), ps, who and maybe a few
  other commands beside, all to no avail.  The next thing I remember was
  being at my terminal again (a multi-window graphics terminal), and
  typing
          cd &#x2F;
          echo \*
  I owe a debt of thanks to David Korn for making echo a built-in of his
  shell; needless to say, &#x2F;bin, together with &#x2F;bin&#x2F;echo, had been
  deleted.  What transpired in the next few minutes was that &#x2F;dev, &#x2F;etc
  and &#x2F;lib had also gone in their entirety; fortunately Neil had
  interrupted rm while it was somewhere down below &#x2F;news, and &#x2F;tmp, &#x2F;usr
  and &#x2F;users were all untouched.
  
  Meanwhile James had made for our tape cupboard and had retrieved what
  claimed to be a dump tape of the root filesystem, taken four weeks
  earlier.  The pressing question was, &quot;How do we recover the contents
  of the tape?&quot;.  Not only had we lost &#x2F;etc&#x2F;restore, but all of the
  device entries for the tape deck had vanished.  And where does mknod
  live?  You guessed it, &#x2F;etc.  How about recovery across Ethernet of
  any of this from another VAX?  Well, &#x2F;bin&#x2F;tar had gone, and
  thoughtfully the Berkeley people had put rcp in &#x2F;bin in the 4.3
  distribution.  What&#x27;s more, none of the Ether stuff wanted to know
  without &#x2F;etc&#x2F;hosts at least.  We found a version of cpio in
  &#x2F;usr&#x2F;local, but that was unlikely to do us any good without a tape
  deck.
  
  Alternatively, we could get the boot tape out and rebuild the root
  filesystem, but neither James nor Neil had done that before, and we
  weren&#x27;t sure that the first thing to happen would be that the whole
  disk would be re-formatted, losing all our user files.  (We take dumps
  of the user files every Thursday; by Murphy&#x27;s Law this had to happen
  on a Wednesday).  Another solution might be to borrow a disk from
  another VAX, boot off that, and tidy up later, but that would have
  entailed calling the DEC engineer out, at the very least.  We had a
  number of users in the final throes of writing up PhD theses and the
  loss of a maybe a weeks&#x27; work (not to mention the machine down time)
  was unthinkable.
  
  So, what to do?  The next idea was to write a program to make a device
  descriptor for the tape deck, but we all know where cc, as and ld
  live.  Or maybe make skeletal entries for &#x2F;etc&#x2F;passwd, &#x2F;etc&#x2F;hosts and
  so on, so that &#x2F;usr&#x2F;bin&#x2F;ftp would work.  By sheer luck, I had a
  gnuemacs still running in one of my windows, which we could use to
  create passwd, etc., but the first step was to create a directory to
  put them in.  Of course &#x2F;bin&#x2F;mkdir had gone, and so had &#x2F;bin&#x2F;mv, so we
  couldn&#x27;t rename &#x2F;tmp to &#x2F;etc.  However, this looked like a reasonable
  line of attack.
  
  By now we had been joined by Alasdair, our resident UNIX guru, and as
  luck would have it, someone who knows VAX assembler.  So our plan
  became this: write a program in assembler which would either rename
  &#x2F;tmp to &#x2F;etc, or make &#x2F;etc, assemble it on another VAX, uuencode it,
  type in the uuencoded file using my gnu, uudecode it (some bright
  spark had thought to put uudecode in &#x2F;usr&#x2F;bin), run it, and hey
  presto, it would all be plain sailing from there.  By yet another
  miracle of good fortune, the terminal from which the damage had been
  done was still su&#x27;d to root (su is in &#x2F;bin, remember?), so at least we
  stood a chance of all this working.
  
  Off we set on our merry way, and within only an hour we had managed to
  concoct the dozen or so lines of assembler to create &#x2F;etc.  The
  stripped binary was only 76 bytes long, so we converted it to hex
  (slightly more readable than the output of uuencode), and typed it in
  using my editor.  If any of you ever have the same problem, here&#x27;s the
  hex for future reference:
          070100002c000000000000000000000000000000000000000000000000000000
          0000dd8fff010000dd8f27000000fb02ef07000000fb01ef070000000000bc8f
          8800040000bc012f65746300
  
  I had a handy program around (doesn&#x27;t everybody?) for converting ASCII
  hex to binary, and the output of &#x2F;usr&#x2F;bin&#x2F;sum tallied with our
  original binary.  But hang on---how do you set execute permission
  without &#x2F;bin&#x2F;chmod?  A few seconds thought (which as usual, lasted a
  couple of minutes) suggested that we write the binary on top of an
  already existing binary, owned by me...problem solved.
  
  So along we trotted to the terminal with the root login, carefully
  remembered to set the umask to 0 (so that I could create files in it
  using my gnu), and ran the binary.  So now we had a &#x2F;etc, writable by
  all.  From there it was but a few easy steps to creating passwd,
  hosts, services, protocols, (etc), and then ftp was willing to play
  ball.  Then we recovered the contents of &#x2F;bin across the ether (it&#x27;s
  amazing how much you come to miss ls after just a few, short hours),
  and selected files from &#x2F;etc.  The key file was &#x2F;etc&#x2F;rrestore, with
  which we recovered &#x2F;dev from the dump tape, and the rest is history.
  
  Now, you&#x27;re asking yourself (as I am), what&#x27;s the moral of this story?
  Well, for one thing, you must always remember the immortal words,
  DON&#x27;T PANIC.  Our initial reaction was to reboot the machine and try
  everything as single user, but it&#x27;s unlikely it would have come up
  without &#x2F;etc&#x2F;init and &#x2F;bin&#x2F;sh.  Rational thought saved us from this
  one.
  
  The next thing to remember is that UNIX tools really can be put to
  unusual purposes.  Even without my gnuemacs, we could have survived by
  using, say, &#x2F;usr&#x2F;bin&#x2F;grep as a substitute for &#x2F;bin&#x2F;cat.
  
  And the final thing is, it&#x27;s amazing how much of the system you can
  delete without it falling apart completely.  Apart from the fact that
  nobody could login (&#x2F;bin&#x2F;login?), and most of the useful commands
  had gone, everything else seemed normal.  Of course, some things can&#x27;t
  stand life without say &#x2F;etc&#x2F;termcap, or &#x2F;dev&#x2F;kmem, or &#x2F;etc&#x2F;utmp, but
  by and large it all hangs together.
  
  I shall leave you with this question: if you were placed in the same
  situation, and had the presence of mind that always comes with
  hindsight, could you have got out of it in a simpler or easier way?
  Answers on a postage stamp to:
  
  Mario Wolczko</code></pre>