Requirements for integration of DEWS technology with THREDDS and GeoServer

Why integrate technologies?

• Make a smaller number of better-quality products (avoid fragmenting effort)
• Put DEWS code under better governance for long-term benefit

Some terminology clarification

In this project we are only concerned with gridded data, such as numerical model output and satellite imagery. "Vector" or "point" data are outside the scope of this project.

For the purposes of this project a "Dataset" is either a single NetCDF file, an aggregation of NetCDF files (e.g. a number of timesteps from a computer simulation) or an OPeNDAP endpoint. In the Java NetCDF libraries, a dataset is represented by the NetcdfDataset class.

Metadata in the NetCDF files can be assumed to conform with the Climate and Forecast (CF) conventions. Other conventions are outside the scope of this project.

A Dataset contains a number of Variables, which are the quantities that we're interested in (temperature, salinity, velocity etc). In WCS, a variable is modelled as a Coverage. In WMS, a variable is modelled as a Layer. Therefore, when a Dataset is added to THREDDS or GeoServer, a number of Coverages/Layers will be created.

What features of DEWS technology do we need to retain?

Configuration requirements

• Need the ability to add a Dataset to the catalogue. A Dataset is identified by a single string (representing the path to the NetCDF file, the path to an aggregation file or an OPeNDAP URL).
• Need the ability to specify the data reading code that will be used for the Dataset. (It's possible that we'll be able to create a generic CF-NetCDF reader but I think it's a good idea to retain some flexibility to accommodate datasets that nearly, but don't quite, fit in with CF, perhaps because CF has not caught up with requirements.)
• System admins should not have to add any other configuration information: all the metadata should be in the NetCDF file headers.
• Should be auto-updatable without manual intervention through the UI or otherwise (support for real-time data).

Data reading requirements

• Need to be able to read data from a number of horizontal coordinate systems of the data (defined by CF)
  • Rotated-pole
  • Lambert equal-area
  • Polar stereographic
  • ...more...
  • Arbitrary curvilinear (hard but important!)
  • Java NetCDF libraries will do most of the work for us
• Need to be able to read data from a number of vertical coordinate systems of the data
  • Elevation (metres)
  • Pressure
  • Sigma (fractional terrain-following) coordinates
• Create "virtual layers/coverages", e.g. create a velocity field from its northward and eastward components

Data output requirements

• Serve 4-D CF-NetCDF through WMS and WCS
• Ability to generate CF-compliant NetCDF files as WCS outputs
• Ability to serve data in its original coordinate system
  • no interpolation, no reprojection
• Ability to specify custom data-reading code for a particular dataset
  • (Maybe we don't need this but some datasets are peculiar.)

Image generation requirements

• Fast generation of WMS images
  • (see here for a description of how this works in the DEWS WMS).
• Ability to adjust colour scale for images
  • Currently done via vendor-specific parameters in the WMS request
  • Could this be done via SLD?
• Ability to render vector quantities as arrows

What will we need in future?

• Ability to keep up with WCS and WMS developments (especially WCS)
• Asynchronous delivery of large data volumes
  • Client gets a "ticket" and uses this to poll for progress
  • Downloads data when ready
  • See "WCSplus" discussions
• Security
  • Ability to secure certain datasets
  • Ability to log and audit usage by each user

-- JonBlower - 25 Feb 2008