This tutorial explains how to create a new experiment from an existing one by copying the initial files and setting up the experiment directory.
ECMWF HPCF
ECMWF operates a Cray based High Performance Computing Facility (HPCF). It consists of two identical Cray XC40 clusters with their own storage, but with equal access to the high performance working storage of the other cluster. This provides the benefit of having one very large system but the dual clusters add significantly to the resiliency of the system.
Each cluster has 20 cabinets of compute nodes and 13 cabinets of storage. The compute nodes contain two Intel Xeon "Broadwell" processors each with 18 cores. Four compute nodes sit on one blade, sixteen blades sit in a chassis and there are three chassis per frame. This gives a maximum of 192, or 6,912 processor cores per cabinet.
For more details about the ECMWF HPCF, please see: http://www.ecmwf.int/en/computing/our-facilities/supercomputer
Login to ECMWF Cray High Performance Computing Facility (HPCF)
Each group or participant will have a training user account on the ECMWF system, beginning with 'troifs'. This is different from the user account on the classroom computers.
First login to the ECMWF gateway computer. From here you will login to the HPCF.
ssh troifsXX@ecgb06.ecmwf.int # substitute your user id for XX troifsXX@ecgb06.ecmwf.int's password: # give your password when prompted using the securID hardware token (one time password)
Next login to the Cray login nodes:
ssh cca troifs@cca-login2:~>
The contents of the account should look like:
troifs1@cca-login2:~> ls bin make python t21test scratch
If any are missing please let us know.
make: contains the OpenIFS executable for version 40r1. For the purpose of this workshop, the model has already been compiled to avoid delay and adding unnecessary load to the login nodes.
t21test: contains an example low resolution test run for OpenIFS.
Directories bin and python: contains commands and script for use during the workshop.
scratch: is the directory in which we will run OpenIFS and process the output files.
Please do not store large files in your home directory on the gateway or HPCF login. All OpenIFS experiments and large files should be stored under 'scratch'. The command quota can be used to determine available space.
Creating the experiment initial files
This section will explain how to copy and change the experiment id.
Initial files for the observed SST experiment, 'ob00', and the climatological SST experiment 'clim', have been created and are made available in the directory : /perm/rd/openifs/oifs_workshop_2017/expts-inidata/
This directory contains multiple dates for each experiment:
ob00/2015110100 to ob00/2015110100 etc.
Copy previous experiment
Before creating the experiment, we first need to create the initial data. If you plan to rerun the ob00 or clim experiments and use the initial data 'as-is' you can skip this step.
In this example, we will create a new experiment id from the existing ob00 experiment, without changing any of the data itself. You can use your own previous experiments, the only difference is the location of the starting initial data.
Note that we copy all the files needed to start the model, not just the initial data files for the atmosphere and surface (ICMSH* and ICMGG* files), but also the climatology (ICMCL*), the wave model start files and the namelists.
In this example, we want to use a new experiment id, ob01, to distinguish it from ob00.
First step is to make a copy of the ob00 data to your /scratch directory:
cd scratch # starting from your home directory mkdir -p inidir/ob01 # our new experiment id cd inidir/ob01 cp -rL /perm/rd/openifs/oifs_workshop_2017/expts-inidata/ob00/2015110100 . # those last characters are a 'space' and then a 'fullstop', '.' means "here".
Make sure you use the -L option! This ensures any 'linked' files are copied as actual files and not symbolic links (true for the fort.4 file). Not doing so will make the fort.4 uneditable.
Ignore any errors about files not being copies because of permission problems.
cd inidir/ob01 cp -rL /perm/rd/openifs/oifs_workshop_2017/expts-inidata/ob00/2015110[1-5]* .
Please only copy the initial dates you intend to use. Each date uses 0.5 Gbyte of file storage.
cd inidir/ob01 cp -rL /perm/rd/openifs/oifs_workshop_2017/expts-inidata/ob00/201511* . # this will be slow!
Change experiment id
OpenIFS forecasts are identified by an 'experiment id', a four letter string. An experiment could consist of a single forecast date or multiple starting dates. It can be of any length and could also include a restarted forecast. However, an experiment only has one horizontal and vertical resolution.
The experiment id is contained the names of the initial files, and it is encoded into the GRIB messages within those files.
Listing experiment id
The command 'exptid' can be used to check and change the experiment id in the GRIB files.
troifs0@cca-login3:> cd inidir/ob01/2015110100 troifs0@cca-login3:> exptid ICMSHob00INIT In file ICMSHob00INIT, values of key experimentVersionNumber are: ob00
The 'experimentVersionNumber' is the GRIB parameter encoded in the GRIB fields.
This can also be seen using the 'grib_ls' command to list the contents of the file:
troifs0@cca-login3:> grib_ls -p shortName,typeOfLevel,dataDate,experimentVersionNumber ICMSHob00INIT ICMSHob00INIT shortName typeOfLevel dataDate experimentVersionNumber t hybrid 20151101 ob00 t hybrid 20151101 ob00 t hybrid 20151101 ob00 .......
The 'exptid' and 'grib_ls' commands can also be used for multiple files:
troifs0@cca-login3:> exptid ICM* In file ICMCLob00INIT, values of key experimentVersionNumber are: 0001 In file ICMGGob00INIT, values of key experimentVersionNumber are: 0001 ob00 In file ICMGGob00INIUA, values of key experimentVersionNumber are: ob00 In file ICMSHob00INIT, values of key experimentVersionNumber are: ob00
Note that the ICMCL and ICMGG*INIT files both have a experimentVersionNumber key of '0001'. This is used for climatological fields.
Set new experiment id
Use the exptid command to set the new experiment id to 'ob01':
troifs0@cca-login3:> exptid -n ob01 ICM*ob00* Changing expid from 'ob00' to 'ob01' for ICMCLob00INIT and writing to new file ICMCLob01INIT. Changing expid from 'ob00' to 'ob01' for ICMGGob00INIT and writing to new file ICMGGob01INIT. Changing expid from 'ob00' to 'ob01' for ICMGGob00INIUA and writing to new file ICMGGob01INIUA. Changing expid from 'ob00' to 'ob01' for ICMSHob00INIT and writing to new file ICMSHob01INIT.
Verify the command worked by checking the experimentVersionNumber in the new files:
troifs0@cca-login3:> exptid *ob01* In file ICMCLob01INIT, values of key experimentVersionNumber are: 0001 In file ICMGGob01INIT, values of key experimentVersionNumber are: 0001 ob01 In file ICMGGob01INIUA, values of key experimentVersionNumber are: ob01 In file ICMSHob01INIT, values of key experimentVersionNumber are: ob01
To save space, you can delete the ob00 files (these will always be available in the directory where they were copied from)
troifs0@cca-login3:> rm ICM*ob00* rm: remove regular file `ICMCLob00INIT'? y rm: remove regular file `ICMGGob00INIT'? y rm: remove regular file `ICMGGob00INIUA'? y rm: remove regular file `ICMSHob00INIT'? y
Further modification to initial files
Now the initial files with the correct experiment ID have been prepared, further modifications to the input files can be made.
To modify the SST parameter, please see other tutorial.
Creating the forecast experiment
Once the initial data directory is prepared,
Running the forecast experiment
blah blah blah
Postprocessing
To postprocess the OpenIFS output files,
Useful Unix commands
rm -rf 2015110100 rm -rf 2015110[1-5]00
du -hs 2015110100
quota
whicih will produce output similar to the following. The '$SCRATCH' quota is important as this is the total limit for the filesystem /scratch where your experiments are stored.
Quota for $HOME and $PERM:
Disk quotas for user troifs0 (uid 16144):
Filesystem blocks quota limit grace files quota limit grace
cnasa1:/vol/home
226M 480M 500M 132 20000 22000
Disk quotas for user troifs0 (uid 16144):
Filesystem blocks quota limit grace files quota limit grace
cnasa2:/vol/perm
0 26624M 27648M 1 200k 210k
Quota for $SCRATCH ($TEMP) including $SCRATCHDIR ($TMPDIR):
Disk quotas for user troifs0 (uid 16144):
Filesystem kbytes quota limit grace files quota limit grace
/lus/snx11062 767099824 32212254720 32212254720 - 116570 5000000 5000000 -
Disk quotas for group ectrain (gid 1400):
Filesystem kbytes quota limit grace files quota limit grace
/lus/snx11062 767105640 0 0 - 118078 0 0 -