Details of GeoServer integration

This page (and its children) are for synthesizing information concerning the adaptation of GeoServer for 4D CF-NetCDF in the SHAVER project.

Summary of current position

At the time the SHAVER project started (Jan 08) at least two groups had already made significant progress towards solving the problem of serving 4D CF-NetCDF in GeoServer, however not all problems have been solved and not all developments have made it into the "official" GeoServer release. In SHAVER we will work closely with two groups, GeoSolutions and Geomatys, who have made progress along these lines. This is a summary of the current state of play, edited from email conversations (please correct if there are any mistakes here!)

Background

The work to support multidimensional raster was started by Alexander Petkov from USFS FireLab which added initial support for time in WMS requests since they needed it for their WMS servers based on GeoServer. Afterwards, http://www.geomatys.fr/ has improved this capability doing some work which can be found under the unsupported modules coverageio, coverageio-hdf and coverageio-etcdf on geotools trunk (add more info if needed from geomatys guys).

During summer 2007 Daniele Romagnoli did some work in order to eplore how to manage complex sets of multidimensional coverages at the ImageIO level under the google summer of code program having Simone Giannecchini as a mentor. He focused on HDF files as NURC receives from APS and TERASCAN systems, as well as on GriB files that are generated by various MetOc models in Italy and US (FNMOC, LAMI, SWAN, etc..). The plan for this work was to lay down a proposal for GeoTools and GeoServer on how to maangend coverages right from the ImageIO level, and this proposal should have triggered a discussion with the GeoMatys associates, which on their turn were focusing on netCDF, in order to replace the current interfaces for doing coverage I/O in geotools (the "(in)famous" AbstractGridCoverage2DReader hack). The first outcome of this work was a lot of code at the imageio level which can be found inside the https://imageio-ext.dev.java.net/ and http://www.geo-solutions.it/docs/ImageIOMetaData_for_MultiDim-Coverages.pdf document on coverage metadata management, for which not a tremendous amount of feedback was received.

During summer 2007 Simone Giannecchini, received a mandate from http://www.nurc.nato.int to improve the capabilities of GeoServer to perform classification on the fly on raster for both MetOc data (basically applying colormaps) and remote sensing data and their browse images (where contrast enhancement is required). This work was performed in the geotools 2.4.x-rs branch and right now there is a proposal pending to port it to geotools trunk and egotools stable branch 2.4.x-rs for use with GeoServer 1.6.2 and greater and GeoServer 1.7.x. See http://docs.codehaus.org/display/GEOTOOLS/Raster+Symbolizer+support . It is worth to point out that we do NOT assume anywhere to work with RGB data! There have been some xchange of emails where I have underlined that GeoTools coverage modules where designed in a way that actually working with RGB data can be complicated, but that is it. The raster symbolized can work on any type of data.

In order to come up with a prototype that tried to integrate work done from GeoMatys with work done by GeoSolutions, Simone Giannecchini suggested in this http://www.nabble.com/Time-and-Elevation-support-for-WCS-and-WMS-td13390661.html#a13418595 to create a specific branch for this work which was created here https://svn.codehaus.org/geoserver/branches/1.6.x-Cov-nD/ and as of now relies on the 2.4.xrs branch of GeoTools. It integrates the prototypal work for multidimensional grib files as well as some HDF formats, as I said above, the raster symbolizer work and also some more work on watermarking. This prototype was delivered to NURC during the first two weeks of January and it is actually used to ship real-time AIS coverage on the mediterranean sea to JFC Naples as well as a WCS test-server to instruct the NRL web crawlers to search and catalog WCS services and dataset.

As part of this year work we would like to support the postgrid project as a way to add more coverages to geoserver as agreed with Martin Desruisseaux from GeoMatys during FOSS4G07 because as far as I understood they have abandoned the GeoServer and there is no plan to merge their work into it.

-more info from geomatys-

GeoMatys work

The following codes, courtesy of Martin Desruisseaux, demonstrate how we can read variables from a simple netcdf file.

       /*
         * The standard ImageReaders provided in Java accept only input of kind ImageInputStream.
         * Our NetCDF image reader accepts File, URL, URI, String and ordinary InputStream as well,
         * but this is specific to our ImageReader. 
         *
         * We use the UCAR Netcdf library under the hood to read the data file.
         */
        final File input = new File("F:\\workspace\\GeoMatysNetcdfReader\\FOAM_20070411.0.nc");
        reader.setInput(input);
        final boolean IMPLEMENTATION_SPECIFIC = (reader instanceof NetcdfImageReader);
        if (IMPLEMENTATION_SPECIFIC) {
            final NetcdfImageReader ncReader = (NetcdfImageReader) reader;
            String[] variables = ncReader.getVariables();
            System.out.println("Variables:");
            System.out.println(Arrays.toString(variables));
            System.out.println();
            variables = new String[] {"TMP", "U", "V"};
            ncReader.setVariables(variables);
        }
  
       /*
         * Lets suppose that we are interested only in a subarea of the dataset, said 60°W to 80°E
         * and 70°S to 80°N. We can declare this area as the "source region", which must be specified
         * in pixel coordinates. The upper-left corner is (0,0) and the two last arguments in Rectangle
         * construction are width and height. In a future version, we will calculate automatically
         * the pixel coordinates from the geographic area using the above metadata. But it will be
         * possible only when the metadata work will be finished.
         */
        final ImageReadParam param = reader.getDefaultReadParam();
        final Rectangle region = new Rectangle(30, 10, 140, 150);
        param.setSourceRegion(region);
        /*
         * Lets suppose that we are interrested to the values at some depths only. Our NetCDF reader
         * stores the depths as different image bands. It may seems strange at first, but it doesn't
         * clash with any NetCDF feature since NetCDF format has no notion of image band. It is also
         * not that different from wha Sun Microsystems does in Java Advanced imaging for example,
         * when complex numbers are stored with the real part in the first band and the imaginary
         * part in the second band. It also fit naturally with Raster.getSampleDouble(x,y,b) as we
         * will see later. Finally it allow us to avoid the introduction of custom API at this stage.
         *
         * However this approach force us to introduce one departure compared to standard Java API.
         * The Image I/O specification said that if no bands are explicitly defined, the ImageReader
         * should read all of them. In our NetCDF image reader case, if no bands are explicitly
         * defined, only the first depth (or "first band") is read. We do that because NetCDF files
         * typically contains tens depths and we assumed that users usually don't want to load all
         * of them.
         */
        int[] depthLevels = new int[] {2};
        param.setSourceBands(depthLevels);
        /*
         * Other standard parameter options are available: setSourceSubsampling(...) for reading
         * only 1 pixel over 2 or 3, etc. ("stride" in NetCDF terminology), and some more
         * sophesticated options for controlling the image to be created.
         *
         * Now read the image. We read the image at index 0, which map to "Sea Surface Temperature"
         * (see above). Note that the "param" argument is optional. If not provided, the image
         * reader would read the whole region of first depth.
         */
        final BufferedImage image = reader.read(0, param);
        reader.dispose();
        /*
         * The image that we get stores the values as floating points. Actually the image format
         * has been determined automatically from the NetCDF file. It could have been signed or
         * unsigned 16 bits integer as well for instance...
         *
         */
        final int type = image.getSampleModel().getDataType();
        System.out.println("Sample data type is float: " + (type == DataBuffer.TYPE_FLOAT));
    

 

Further information:

• Presentation at FOSS4G 2007: http://www.foss4g2007.org/presentations/view.php?abstract_id=225
• Demo server (now outdated): http://demo6.geomatys.fr/seagrid/demo/ifremer/

GeoSolutions Work

GeoSolutions have developed work along three main lines that are relevant to the SHAVER project:

1. Support for n-D coverages at the level of imageio (https://imageio-ext.dev.java.net/), geotools (branch 2.4.x-rs of GeoTools) and geoserver ( 1.6.x-Cov-nD geoserver branch). We have been concentrating on GriB1 format for MetOc data and HDF for remote sensing data
2. Support for GDAL at the level of imageio (https://imageio-ext.dev.java.net/), geotools (branch 2.4.x-rs of GeoTools, proposal to port to trunk and 1.6.x is here:http://docs.codehaus.org/display/GEOTOOLS/Raster+Symbolizer+support) and geoserver ( 1.6.x-Cov-nD geoserver branch)
3. Work on supporting SLD-compliant Raster Symbolizer for rendering gridded data as WMS images (branch 2.4.x-rs of GeoTools, proposal to port to trunk and 1.6.x is here:http://docs.codehaus.org/display/GEOTOOLS/Raster+Symbolizer+support). This try to lat down a path on how the coverage can be separated from its visualization.
4. Proposal for ImageIO metadata to support multidimensional coverage (see http://www.geo-solutions.it/docs/ImageIOMetaData_for_MultiDim-Coverages.pdf)

.

Here you can find some more references

1. Proposal for ImageIO metadata to support multidimensional coverage (see http://www.geo-solutions.it/docs/ImageIOMetaData_for_MultiDim-Coverages.pdf)
2. FOSS4G07 presentation: http://www.foss4g2007.org/presentations/view.php?abstract_id=241
3. Demo server at NURC (tomorrow )

Discussion

Some of the problems integrating nD Coverage reader with Geoserver:

Geoserver admin interface requires a file for every layer that is created. This does not suit well with n-D coverage where we often have many files for the same layer, in our case one for each date in a time series.

The 1.6.x-Cov-nD branch contains changes to the GeoServer configuration in order to allow users to configure multiple coverage from a single coverage store. At the baseline let's say that modifying the GeoServer in order to do is feasible.

The class that GeoServer uses for reading (AbstractGridCoverage2DReader ) is designed for 2D coverage read from a file. It doesn't fit well for coverage read from a database and for anything with more than one file by layer (e.g. time series). AbstractGridCoverage2DReader is also limited to 2D rasters.

--simone-- As the name states AbstractGridCoverage2DReader allows people to read GridCoverage2D , I don't think it is a secret that it is limited to 2D coverages :-). In the Cov-nD branch we have tried to extend this approach, but anyway there is a proposal pending here http://geotools.codehaus.org/GridAccess+based+on+WCS+Specification, I guess that it would be good to see some contructive feedback there.

The streaming renderer in GeoServer has got some limitations which adversely affect performance and memory usage.

--simone-- I would say that this is a bit vague. The StremingRenderer has some limitations for sure, but a lot of people are using GeoServer WMS to serve coverages that are gigabytes hence I would not state this in such a generic way. To me it seems like saying, java is slow :-).

--aaime-- Can you be more specific? Last time I checked it was designed to avoid reading the whole coverage in memory (provided the coverage can be read using tiling). This is not to say the memory usage could be lowered, on the contrary, it's possible.

Users currently have to specify a lot of information (e.g. geographic envelope, coordinate reference system) in GeoServer admin interface, which in fact they should not need to, because they are available by other means (file header, database, etc.)

--simone-- Most part of this information are infact taken directly from the sources, when they are avaliable. Experience says that many people want to use file where this info are missing hence the GeoServer interface tries hard to give them the possibility to provide them.

-aaime -- it's prefilled with srs, bbox read from the coverage, the user can change that if he needs to, but he's not forced to.

The main drawback with the current GeoApi GridCoverageReader interface is that there is no way to get real information about a raster source unless you instantiate a GridCoverage . As an instance it is impossible to know the envelope, the number of overviews, the tile size. This information is needed in order to perform decimation on reading or to use built-in overviews. This really impacts the ability to exploit raster datasets in a desktop environment where caching is crucial.

I (Simone Giannecchini) have been suggesting to expose more information to users at an erlier stage with respect to what the actal GridCoverageReader interfaces do in order to allow them to fine tune requests before they actually do them (check the gdal model for a comparison or the ImageIO model).

---

See the TechnologyComparison for a summary of the above, and also a comparison with THREDDS and DEWS technology.

Issues outstanding

These issues are in no particular order. Please do not delete an issue unless it is factually incorrect. When the issue is resolved add a comment to the table.

Issue number	Description	Status (if blank, outstanding)
SHAVER-GEO-1	How should NetCDF datasets be registered in GeoServer as data sources? (I.e. what should the catalogue look like?)
SHAVER-GEO-2	How can we render vector quantities in the WMS (e.g. wind barbs, current arrows)?
SHAVER-GEO-3	How can we merge nD GeoTools with GeoServer?
SHAVER-GEO-4	How can we enable adding of datasets (mutiple files per layer, multiple layers per file) via admin interface?

-- JonBlower - 01 Feb 2008