An administrator accidentally deleted the production database
My very first job - ~25 years ago.<p>Destroyed the production payroll database for a customer with a bug in a shell script.<p>No problem - they had 3 backup tapes.<p>First tape - read fails.<p>Second tape - read fails.<p>Third tape - worked.... (very nervous at this point).<p>I think most people have an equivalent educational experience at some point in their careers.<p>Edit: Had a project cancelled for one customer because they lost the database of test results..... 4 months work! Their COO (quite a large company) actually apologised to me in person!<p>Edit: Also had someone from Oracle break a financial consolidation system for a billion dollar company - his last words were "you need to restore from tape" and then he disappeared. I was <i>not</i> happy as it was his attempts at "improving" things were the cause of the incident! Wouldn't have been angry if he had admitted he had made a mistake and worked with us to fix it - simply saying "restore from tape" and running away was not a good approach.