An administrator accidentally deleted the production database
Treating app servers as cattle, i.e. if there&#x27;s a problem just shoot &amp; replace it, is easy nowadays if you&#x27;re running any kind of blue&#x2F;green automated deployment best practices. But DBs remain problematic and pet-like in that you may find yourself nursing them back to health. Even if you&#x27;re using a managed DB service, do you know exactly what to do and how long it will take to restore when there&#x27;s corruption or data loss? Having managed RDS replication for example doesn&#x27;t help a bit when it happily replicates your latest app version starting to delete a bunch of data in prod.<p>Some policies I&#x27;ve personally adopted, having worked with sensitive data at past jobs:<p>- If the dev team needs to investigate an issue in the prod data, do it on a staging DB instance that is restored from the latest backup. You gain several advantages: Confidence your backups work (otherwise you only have what&#x27;s called a Schrödinger&#x27;s-Backup in the biz), confidence you can quickly rebuild the basic server itself (try not to have pets, remember), and an incentive to the dev team to make restores go faster! Simply knowing how long it will take already puts you ahead of most teams unfortunately.<p>- Have you considered the data security of your backup artifacts as well? If your data is valuable, consider storing it with something like <a href="https:&#x2F;&#x2F;www.tarsnap.com" rel="nofollow">https:&#x2F;&#x2F;www.tarsnap.com</a> (highly recommended!)<p>- In the case of a total data loss, is your data retention policy sufficient? If you have some standard setup of 30 days worth of daily backups, are you sure losing a days worth of data isn&#x27;t going to be catastrophic for your business? Personally I deploy a great little tool called Tarsnapper (can you tell I like Tarsnap?) that implements an automatic 1H-1D-30D-360D backup rotation policy for me. This way I have hourly backups for the most valuable last 24 hours, 30 days of daily backups and monthly backups for a year to easily compare month-to-month data.<p>Shamless plug: If you&#x27;re looking to draw some AWS diagrams while Gliffy is down, check out <a href="https:&#x2F;&#x2F;cloudcraft.co" rel="nofollow">https:&#x2F;&#x2F;cloudcraft.co</a> a free diagram tool I made. Backed up hourly with Tarsnap ;)