Zstandard – Fast real-time compression algorithm
We have a delta-encoded binary format for trade logs that we use and zstd still does a pretty good job on them for the last bit.It's just a pity that storage providers won't implement transparent compression. We store our logs on S3 and on-prem and it's just annoying that we have to store them compressed on S3. Would be cool to just store without the compression stage and have S3 do that bit.I haven't tried btrfs with the transparent zstd compression, but perhaps worth a shot.