PyFileFixity: Helping long term storage of data via redundant ECCs
I really like the idea behind storing error-correction codes at the application level.  Disks should just store and retrieve a chunk of bits and it should be pushed up to the application level to decide whether or not those bits were what we wrote.  From there, we can make the right decisions about how durable we really want a file to be.<p>Instead, people seem to have invested a lot of effort in going the other direction; let the application have less control and let many magic layers in the middle figure out the details.  We have had filesystems for decades, which pretend to map names to a list of durable block numbers.  Of course, the blocks are not so durable (nor is the database), so people lose their data all the time and act like it&#x27;s normal.<p>People have a solution to that problem, though, make the block devices more durable!  We can just add software or hardware RAID on top of a bunch of disks, and now suddenly what the filesystem thinks is just one block is now 3 blocks sharded across 3 disks.  Some magical curator process then tells me when disks have failed (as though it can detect that with 100% reliability), and I pop in a new drive and magically everything has been repaired.  Of course, in reality, the entire disk doesn&#x27;t just disappear all at once; some blocks just return data with a flipped bit, or all 0s, or an occasional error.  My experience in running RAID-1 arrays under Linux (for my workstation) is that I notice some errors in the logs and have to manually figure out which disk contains the correct block (this is why I always use 3 disks) and then dd the block from the good device to the degraded device.  By the time the automatic curator shows up to detect and repair this problem for me... all three disks have been RMA&#x27;d.  It is awful and I&#x27;d be surprised to hear a story where someone says &quot;one of my disks failed, I was notified in a timely manner, and my array was repaired when I inserted a new disk&quot;.  It just doesn&#x27;t track enough data to actually work, because it ignores the important of error-correction codes.  (Don&#x27;t get me started on RAID-5.  That has some parity, but not Reed-Solomon codes, and the parity just makes it unlikely that you could ever manually get the array out of a bad state.)<p>But even RAID isn&#x27;t reliable enough, and people don&#x27;t use it, so of course the disk pretends it is more durable than it actually is.  Spinning rust disks have some spare sectors.  If their internal heuristics that you cannot tune or read the source code for think something is wrong, it will store some data saying &quot;now block 42 is actually stored at block 2389473&quot; and my RAID controller, filesystem, and application never have to know!  Hooray!  SSDs are even more egregious; they amplify writes, they garbage collect, they compute proper R-S codes over the data, and have a multi-core CPU running god-knows-what code handling it all.<p>I think it is all crazy.  In my dream world, an SSD would just be a bunch of flash-chips that are not very durable due to the nature of how TLC NAND flash works.  Then my application can decide &quot;this is an important photograph, so I&#x27;m going to shard it across 8 chips on two different controllers and use error-correction codes that can correct 100% of bit errors&quot;.  But, that does not exist.<p>It all makes me mad because when I worked at Google I actually had control, at the application level, over all this.  There was the underlying disk API that you could use directly that just stored a block, with the durability of a datacenter disk drive.  Then on top of that were filesystems and databases where you had to pick the durability for every operation; you could pick between straight redundancy (write to 3 disks) or whatever level of Reed-Solomon correction you wanted (all these are tradeoffs between read speed, write speed, and actual durability).<p>Meanwhile in the real world, people build crazy things that still lose their data.  ZFS loses data.  Cloud block stores lose data.  Consumer databases lose data.  I know photographers and videographers that have lost all their work because of disk failure... they had multiple disks, but their applications did not uses the disks efficiently in a way that could detect and correct errors.  So you come back a year later, and opening your file just says &quot;sorry, error!&quot; and you accept your loss and move on.  Someday, I hope this can stop.  I never want to watch a YouTube video like this again: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=gSrnXgAmK8k" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=gSrnXgAmK8k</a> (LTT&#x27;s Windows-based fileserver with hardware RAID fails randomly and cannot be fixed; all their work gone)