ZSTD 1.5.5 is released with a corruption fix found at Google
> corruptions are detected thanks to the checksum<p>That's good to know because, for important things, I test the archive before throwing the original data away.<p>Not to single out zstd but it's a good opportunity to be reminded: if you make backups, test your backups! A bug like this can also be introduced, not only fixed. I'm not saying it's likely, or that you're likely to be affected by it, but for things you care about, just opening the backup once a year and spot checking some files is not a lot of work for a lot lower risk compared to never checking it. Most backup tools will also let you do checksum validation in an automated fashion, but I prefer to also manually open it and check that it truly works (not that a partial archive has no errors, for example).<p>Anyway, details of the bug are here: <a href="https://github.com/facebook/zstd/pull/3517">https://github.com/facebook/zstd/pull/3517</a><p>> This [affects] the block splitter. Only levels using the optimal parser (levels 13 and above depending on the source size), or users who explicitly enable the block splitter, are affected.<p>So if you use, for example, zstd -9 (I didn't know it went higher, at least not at somewhat reasonable speeds) or below, then you should always have been fine unless you called the block splitter yourself explicitly (or, perhaps, if your backup software does that for you? It sounds like something relevant for deduplication but I'm not sure what this feature is exactly).<p>> The block splitter confuses sequences with literal length == 65536 that use a repeat offset code. It interprets this as literal length == 0 when deciding the meaning of the repeat offset, and corrupts the repeat offset history. This is benign, merely causing suboptimal compression performance, if the confused history is flushed before the end of the block, e.g. if there are 3 consecutive non-repeat code sequences after the mistake. It also is only triggered if the block splitter decided to split the block.<p>If I understand it correctly, the bug triggers under circumstances where the data causes the splitter to split at exactly 2^16 and then doesn't flush the block, and one example where it doesn't do that is if any part of the next 2^17 bytes (128K) is compressible? Not sure what a "repeat code sequence" is, my lz77-geared brain thinks of a reference that points to an earlier repetition, aka a compressed part.