Advanced Scientific Data Format
What timing! I&#x27;m just experimenting with some bioinformatics codes, and wow... the formats are <i>terrible</i>. They&#x27;re mostly sequential text files, some tens of gigabytes in size.<p>Simply uncompressing a file can be a significant bottleneck, as this can inherently use only 1 CPU core.<p>Keep in mind that these files are <i>intended</i> to be processed on monstrously huge 128 core and 2TB memory machines! What happens is that the system uses 0.5% of its capacity to manipulate the files while the other 99.5% is heating the data center air.<p>I&#x27;m looking at the CPU usage metric graph now, and the machine is spending half the time at 100% load and half the time at 1% load. That&#x27;s half the total capacity wasted.<p>If you find yourself in 2022 or later designing a file format intended for bulk data and you use any of the words &quot;stream&quot;, &quot;serialization&quot;, or &quot;text&quot;, stop. Rethink what you have done, and this time consider that normal machines you can buy for normal money have 256 hardware threads. Soon, 512, and then 1024 in just a few years!<p><i>Everything</i> at this scale should be split into blocks for parallel processing. Shoving text files into a .tar.gz archive is <i>not acceptable</i> any more. I don&#x27;t care what 1970s &quot;standard&quot; it adheres to, that just doesn&#x27;t matter any more, the hardware has moved on.<p>I think it&#x27;s high time that the industry standardised on a generic &quot;container&quot; format to replace legacy archive file formats. Something akin to a random-access archive file like zip, but designed so that even a single file can be decoded in parallel, <i>even if compressed</i>.<p>As an example of &quot;parallel thinking&quot;, CRC or SHA style checksums are automatically a no-go. They&#x27;re inherently sequential. Instead, Merkle trees would have to be used.<p>Compression efficiency of small files with random access could be improved by using something like zstd as the internal compression algorithm, but with a shared dictionary stored separately. This would retain the advantages of &#x27;tar&#x27; without requiring sequential decoding of files.<p>Etc...