Alan's tasks for GCEP

SORRY, THIS DOCUMENT IS RATHER BEHIND REFLECTING REALITY. HASSLE ME IF I DON'T UPDATE IT SOON. ALAN (7 JAN 2008)

Note: the diagnostics package is the main piece of work at the moment; other items will be after that is completed, unless I get little bits of time for them.

Diagnostics package

• Aim: a tool to generate easily a standard set of plots for any given run -- ideally from a single command -- so that plots gets produced systematically. Most (all?) of these plots are timeseries of various indices. Should also be able to plot ensemble means and spread. Another requirement is that the package should still produce as much as possible if certain fields are missing.

• Implementation: will be scripts run in python using CDMS to read in UM / PP files, and using R (via the RPy interface) to do the plots. There will be two high-level scripts (possibly run from a single master script): one to calculate the indices and another to do the plotting. The indices calculated should be written out to a netCDF file, allowing for plots of the same indices to be made from other programs if required. If a run is extended, there should be an option to use a previously generated netCDF file as input, and not calculate the indices for the whole run but just the period necessary to append to the previous timeseries. The plotting script should either be able to read indices from a set of netCDF files constituting an ensemble, or should support the reading of ensemble mean, standard deviation and possibly max and min, from files produced from the individual netCDF files by using my ncensemble tool.

• Sub-tasks:
  • Installation of CDAT, R, Rpy on pegasus / gorgon, and document.
    • Status
      • All installed on pegasus already, but cdscan not working. Need to get cdscan working cdscan now working.
      • Some installation documentation in a private file; need to give to Dan, and put usage instructions on the wiki.
      • Once fully working, need to get installed on gorgon too (to avoid excessive NFS use w data on gorgon). (Dan can do this, a useful test of my having documented it properly).
  • Write routines to read UM files and evaluate the following indices and dump to netCDF:
    • general architecture: DiagnosticsPackageArchitecture
    • List of diagnostics:
      • from AtmosIndices
        • GlobalMeanSurfaceTemperature
        • GlobalMeanSurfaceTemperatureAt1.5m
        • Southern Oscillation Index (SOI)
        • NorthAtlanticOscillation (NAO) - difference gridpoint values (Azores minus Greenland-Iceland) maybe check exact(??) [I use pressure at (y=37.5N,x=26W) minus pressure at (y=60N,x=34W), this mimics where HadCM3 has its NAO. You could consider two area averages as well - Leon]
        • Radiation Buget at the top of atmosphere (OLR and total)
          • Need normal and clear sky
          • Also shortwave (include incoming SW to check forcing)
          • global only (even though Leon has latitude bands for some diags)
          • Need to check with Rowan on best definitions to use for SOI, NAO
      • P-E for various regions, including: Antarctica, arctic basin (ask Debbie for other regions)
      • from OceanIndices
        • GlobalMeanSST
        • GlobalMeanSSS
        • WarmWaterVolume (WWV)
        • Heat Content:
          • NB Leon has supplied info from Doug about heat content diags. [I would suggest 360m and 1200m as depths to integrate to, I have algorithm - Leon]
          • [North Atlantic heat content would be interesting, especially split into the gyres: 20N:50N and 50N:66N - Leon]
        • MeridionalOverturningCirculation (MOC) - Atlantic [Averaged over 27.5N-32.5N has a good correlation with heat transport. 40N is also interesting - Leon]
        • Antarctic circumpolar current volume transport - maybe evaluate from streamfunction? (Ask MO?)
        • HeatTransport
        • NINO
          • NINO1+2
          • NINO 3
          • NINO 3.4 (AlsoComparisonWithWWV)
          • NINO 4 [Isn't just nino3.4 enough? - Leon]
        • [There are two interesting SST indices you could consider
          • Tropical Atlantic Dipole: (y=0N:20N@ave,x=77W:10E@ave)-(,y=20S:0S@ave,x=52W:15E@ave)
          • Interdecadal Pacific Oscillation (Copsey-style): (x=170w:100W,y=15N:15S)-0.5*((x=170E:140W,y=25N:40N)+(x=150E:160W,y=40S:25S)) - Leon]
        • Sea Ice
          • Southern and Northern hemisphere sea ice cover area and volume
          • The volume ice export at the 79deg N section (Fram Strait, integrated from 20W-10E) - want a figure in kg/s using density hard-coded in model.
        • Southern Annual Mode (SAM) index (40S-65S, zonal average of sea level pressure, normalised)
        • land surface diagnostics - Need to check with Debbie
          • Snow covered area
          • Mean SWE / snow mass
          • Soil Moisture
        • Need to check written notes taken 29th May for other diagnostics and enter here
    • Status: in progress
  • Write routines to plot indices from netCDF files:
    • Status: pending
  • Identify unnecessary diagnostics.
    • The standard runs that people are using are writing a lot of diagnostics. Many of these may not be needed to calculate the indices listed above. Obtain a list of diagnostics which are actually required, and advise on ones which can be dropped. Create an example job containing a pruned set of diagnostics; people can then merge it into their jobs using the copy_stash script.
    • Status: pending [Chunlei and I have "reduced STASH" versions you could look at - Leon]

Ancillary files

• Aim: There is a problem at the moment, that as the time-dependent forcings runs progress, there are different ancillary files needed for different stages of the runs, and runs have to be restarted manually with different ancillary files. Given the number of ensemble members, this is tedious and increases the chance of mistakes. It would be good to have an automated procedure to use different forcings files for different model dates. Doug has provided some code which may be of assistance but we need to check this.
• Subtasks:
  • Understand what Doug has sent, and whether it meets our needs. Update: Leon has done this already, see UpdatingAncillaries.
  • If necessary, code up some procedure myself. This might involve using the timed script release functionality in the UM. An example methodology would be to have the UM write the model time to a temporary file, and have the script read in the model time and create a symlinks pointing to appropriate ancillary files. The symlinks would then be configured as the filenames of the ancillary files in the UMUI.
  • Produce an example run in the UMUI which makes use of this procedure.
  • Test it
• Status: pending

HadCM3 / Condor

• Aim: Keith would like to know whether we can run HadCM3 (single processor) on the machines in the Condor pool
• Subtasks:
  • Investigate memory requirement of 1x1 job as running on Gorgon, and compare with hardware specs of typical machines
  • Provided this does not take too much time:
    • familiarise myself with Condor by trying a "hello world" Condor job as per info supplied by Dan.
    • having done so, advise Dan / James whether I can anticipate any particular difficulties / challenges / recommended methodology as regards getting the UM running in the framework of Condor
  • In any event, leave the actual implementation (if any) to James -- Keith doesn't want me to spend too much time on this
• Status: pending

Investigate mounting data from BDAN on gorgon

• Aim: for data assimilation runs, Chunlei would like to access ECMWF re-analyses that we think Paul has put onto the Reading BDAN brick. It would be best to mount these using Terragrid.
• Subtasks:
  • With Chunlei, check that the BDAN brick really has the data that is required
  • Check whether the terragrid client (kernel module) is available for the kernel version in use on gorgon, and if so, whether Dan is happy to install it.
    • If so, download it and coordinate with Dan regarding the installation and mounting.
    • If not, options are: request change to kernel version (from Clustervision via Dan), or request support for kernel actually used (with Workstations UK), or just use NFS.
    • Ensure that whatever happens, the filesystem is made safe against rsync deletion accidents(!!!) even if it can't be explicitly mounted read-only.
• Status: pending

Improve ncensemble tool:

• Aim: ncensemble tool will calculate mean and standard deviation. It would be nice to make it do max and min as well. Potential use would be for processing netCDF files containing indices for sake of plume plots etc.
• Status: pending

Archiving of standard PP channels

• Aim: diagnostics output to standard (non-climate-mean) PP channels e.g. o.pd, a.pc, end up in $DATAM rather than um_archive, which is less convenient. Want to move these automatically to um_archive if possible.
• Status: pending

Science runs:

• Aims: perform ensemble runs to complement other work on GCEP
• Subtasks:
  • Meetings with others to discuss details more fully
  • Further actions as defined after meetings
• Status: pending. Nothing expected until August.