I just wrote a blogpost on netCDF compression. Properly leveraging compression can reduce the size of your in/output data, and speed up I/O of your model! Thought it might be interesting for people here.
If anyone has any questions I’d be happy to answer them.
Thanks, that’s a really nice blog post! I’ve got to admit that compression is rarely something I bother with, though I think some of my datasets could definitely benefit from it…
Do you (has anyone) got any experience with NetCDF compression in Fortran? I haven’t looked into it, but from memory I can’t remember any “easy win” solutions like "zlib": True in xarray.
I don’t have any experience with that, however from the netCDF Fortran documentation it seems like it should be straightforward. Although the docs state that it can’t do parallel writing with deflate, and the docs only mention deflate and not zlib.
However, I did find a presentation stating that netCDF >=4.7.4 supports parallel compressed I/O and zstandard/quantization, configured per variable with;
f90_def_var(ncid, VAR1_NAME, NF90_FLOAT, dimids, varid1, &
zstandard_level = 4, shuffle = shuffle)
see https://epic.noaa.gov/wp-content/uploads/2023/08/UIFCW-2023-Tue-10.-Hartnett.pdf
Nice, thanks!
It looks like the official docs have some catching up to do 