Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
This analysis is simplistic.<p>Correlated failures are common in drives.  That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc.<p>When calculating your chance of failure, you <i>have</i> to include that, or your result is bogus.<p>Eg.  Model A of drive has a failure rate of 1% per year, <i>but</i> when failed the symptom is failure of the drive to spin up from cold, however if already spinning it will keep working as normal.<p>3 years later, the datacenter goes down due to a grid power outage and a dispute with diesel suppliers so the generators go down.  It&#x27;s a controlled shutdown, so you believe no data is lost.<p>2 days later when grid power is back on, you boot everything back up, only to find out that 3% of drives have failed.<p>Not a problem.  Our 17 out of 20 redundancy can recover up to 15% failure!<p>However, each customers data is split into files around 8MB, which are in turn split into the 20 redundancy chunks.  Each customer stores say 1TB with you.  That means each customer has ~100k files.<p>The chances that you only have 16 good drives for a file is about (0.97^16 * 0.03^4)<i>20</i>19*18 = 0.3%<p>Yet your customer has 100k files!  The chance they can recover all their data is only (1-0.003)^100000...   Which means every customer suffers data loss :-(