-- AditSantokhee - 16 Sep 2010

Introduction

NetCDF-4 classic model files that use compression can be smaller than the equivalent netCDF-3 files, so downloads are quicker.

The latest netCDF 4.1.1 library provides a new function called nccopy which can be used to convert netCDF-3 files to netCDF-4. But nccopy cannot yet compress variable from ones in a netCDF 3 file since it has no 'deflate' attribute. This capability will be added in an upcoming release of the library. So for time being, we use a useful extension to the nccopy utility that provide compression and other capabilities not available in the original code. The code is available at http://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2010/msg00271.html

In order to compile the program, we need to install the latest netCDF library with ability to create netCDF-4 files (http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-install/Quick-Instructions.html#Quick-Instructions).

Installing netCDF library

As prerequisites, we require the HDF5 1.8.4-patch1 release as well as the zlib compression library, version 1.2.3 (or better). Both of these packages are available from the netCDF-4 ftp site at ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.

1. zlib

sudo mv zlib-1.2.3.tar.gz /usr/local

sudo tar -xvzf zlib-1.2.3.tar.gz

cd zlib-1.2.3

sudo ./configure

sudo make check install

2. HDF5

sudo mv hdf5-1.8.1.tar.gz /usr/local sudo tar -xvzf hdf5-1.8.1.tar.gz

cd hdf5-1.8.1

sudo ./configure --prefix=/usr/local --with-zlib=/usr/local --disable-shared

sudo make install

Make sure to run “make check” for the HDF5 and zlib distributions. They are very well-behaved distributions, but sometimes the build doesn't work (perhaps because of something subtly misconfigured on the target machine). If one of these libraries is not working, netCDF will have serious problems. If “make check” fails for either zlib or HDF5, the problem must be resolved before the netCDF-4 installation can continue.

Upon successful installation of these 2 libraries, we can now install the netCDf library:

3. netCDF

sudo mv netcdf-4.1.1.tar.gz /usr/local

sudo tar -xvzf netcdf-4.1.1.tar.gz

cd netcdf-4.1.1

sudo ./configure --enable-netcdf-4 --with-hdf5=/usr/local -with-zlib=/usr/local

sudo make check install 

The library is built with the following features:

[resc@dexter n4]$ nc-config --all 

--cc -> cc 

--cflags -> -I/usr/local/include 

--libs -> -L/usr/local/lib -lnetcdf -L/usr/local/lib -lhdf5_hl -lhdf5 -L/usr/local/lib -lz -lm 

--cxx -> c++ 

--has-c++ -> yes 

--fc -> 

--fflags -> /usr/local/include 

--flibs -> -L/usr/local/lib -lnetcdf -L/usr/local/lib -lhdf5_hl -lhdf5 -L/usr/local/lib -lz -lm 

--has-f77 -> no 

--has-f90 -> no 

--has-dap -> no 

--has-nc2 -> yes 

--has-nc4 -> yes 

--has-hdf5 -> yes 

--has-hdf4 -> no 

--has-szlib -> no 

--prefix -> /usr/local 

--includedir-> /usr/local/include 

--version -> netCDF 4.1.1 

Compile utility to compress netCDF3 files to netCDF4

We compile the utility to do compression using the --cflags and --libs flags as follows:

gcc -o ncconvert ncconvert.c nciter.c -I/usr/local/include -L/usr/local/lib -lnetcdf -lhdf5_hl lhdf5 -lz -lm

This extension provides the following capabilities:

./ncconvert [-k n] [-m n] infile outfile 

[-k n] kind of netCDF format for output file, default same as input 

1 classic, 2 64-bit offset, 3 netCDF-4, 4 netCDF-4 classic model 

[-m n] size in bytes of copy buffer 

[-d n] deflate vars (n: deflate level [0-9]) 

[-s] shuffle vars 

[-u] remove unlimited dims 

infile name of netCDF input file 

outfile name for netCDF output file 

The the d,s and u options are only valid for netCDF format (-k 3 or -k 4)