Finally I'm starting to look at the possibilities of rewriting GRIB files into NetCDF/CF format. I am going to use the NCEP model data from the IDD as the target datasets, and explore the issues that arise in converting them to netCDF-4.
One of the strengths of GRIB is its ability to store data compactly. The NCEP data mostly uses bit packing and JPEG-2000 wavelet compression. For each 2D slice to store, the range (min, max) of the data along with the number of bits to store (N) determines a scale and offset, which is used to convert the floating point value to an N-bit integer. I would guess that N is chosen for each variable based on the known precision of the variable, although in principle N can be different for each 2D slice of data. Because the scale/offset can be different for each 2D slice of data, this algorithm works well for atmospheric data where there can be large differences in magnitude of a variable at different vertical levels. The scale/offset used by the netCDF classic model, in contrast, must be applied to the entire data, and therefore is less accurate. Effectively GRIB gives us reletive precision ("4 decimal points of accuracy") and netCDF scale/offset gives us absolute precision ("accurate to .01 meters").
So the question is, if we rewrite GRIB into netCDF-4 what size are the netCDF files reletive to GRIB? The standard compression algorithm for NetCDF-4 / HDF5 is the deflate algorithm, also used by gzip, zip, png, HTTP, etc. Deflate is a lossless dictionary encoder, which looks for patterns in the data and stores common patterns efficiently. A completely random set of numbers won't have patterns, and so will not compress. Floating point data can be difficult to compress because the lower bits of the mantissa are essentially random. However, the GRIB data already has those random bits removed when its converted from floating point to N-bit integers. So there's hope that deflate might do a good job on GRIB data.
To start I will use netCDF-4 "out of the box", and see what compression ratios can be achieved, and then investigate what, if anything, can be done to make netCDF files as small as GRIB.
The results show that shuffle makes the file about 10% bigger (3.5/3.2 = 1.1). As the deflate level goes from 1 to 4, the file gets about 10% smaller, at a cost of about 5% extra time (remember this includes the entire copy). Deflate levels greater than 4 don't seem to make the file much smaller for the extra time spent compressing (level 9 was off the chart).
In conclusion, for these kinds of files at least, don't use the shuffle filter and use a deflate level of 1-4.
Next time: netCDF-4 vs GRIB file sizes.