Each participant will have a training user account on the ECMWF Cray system. This is different from the user account on the classroom computers.
Begin by logging onto the Cray login nodes:
ssh trcrayXX@ccb <<< use your Cray user id Password: xxxxxxxx trcray@ccb-login2:~> |
The contents of the account should look like:
trcray1@ccb-login2:~> ls bin modules.sh oifs38r1 scratch |
If any are missing please let us know.
This directory contains the OpenIFS source code and OpenIFS executable.
For the purpose of this workshop, the model has already been compiled to avoid delay and adding unnecessary load to the login nodes.
Examine the files in the oifs38r1 directory. |
This is the directory in which we will run OpenIFS.
Each time you login and before doing any work in this account, make sure you have the correct environment loaded to run and work with OpenIFS.
|
This will load the appropriate modules for OpenIFS and set the choice of compiler and build environment. It is good practise to use something similar for your own OpenIFS installation.
An ensemble experiment consists of a 'control' and multiple 'perturbed' forecasts, or 'ensemble members'. For the operational ECMWF ensemble forecast, 50 members are used.
For this exercise, to save computer time, each participant will run a single, different 'ensemble member' and we will create the complete forecast ensemble from the individual ensembles run by the class as a whole. A later exercise will show how to do this from the model output GRIB files.
The first exercises show how to create the experiment directories, prepare the jobs and submit them to the ECMWF Cray.
Use the command createENS to create the ensemble experiment:
createENS --help
usage: createENS [-h] -d <startdate/enddate/hr_inc> [-m <members>]
Set up ensemble forecast for the OpenIFS 2015 workshop
optional arguments:
-h, --help show this help message and exit
-d <startdate/enddate/hr_inc>, --date <startdate/enddate/hr_inc>
List of date(s) to retrieve, format: YYYYMMDDHH. The
only dates available are: 2013102400/2013102700/24
-m <members>, --members <members>
Number of ensemble members to create e.g. 3 (default 1) |
Decide which of the available dates to use. Use the createENS command to create an ensemble with just a single member. |
Change to the experiment directory labelled by the date chosen:
There are two experiment directories: cf000 and pf001. cf000 is for the control forecast which is not used here. pf001 is the single perturbation forecast.
Understand what the different files in the experiment directory are for. ICM* : these are the initial starting files. ICMGG* are the initial gridpoint files, ICMSH* are the initial spectral fields. Use the 'grib_ls' command to examine the initial files:
The file: namelistfc is the model 'NAMELIST'. It contains a list of variable settings or 'switches' that control what the model does. These variables are grouped into separate fortran namelists. |
In IFS, each ensemble member uses the stochastic physics schemes to generate uncertainty. A random number 'seed' is used by the stochastic scheme to generate a different forecast. This random number seed is changed by altering the ensemble member value in the model's namelist. Each ensemble member must have a unique number and therefore random number seed, in order to produce a different forecast.
Make a copy of the namelist file and edit it to set the ensemble member value to the one you are given. Make sure you are in the 'pf001' directory.
and make the following changes to the namelist variables:
The variable NENSFNB is important as this determines the random number seed by altering the ensemble member value.
|
The next steps are to create and submit a batch job to run on the ECMWF Cray XC30 system.
To create the batch job, use the oifs_run command. This creates a small batch job file ready to submit.
oifs_run -e exptid [-m path-to-master.exe] [-r resolution] [-l namelist] [-s timestep] [-n nproc] [-f fcast len] [-x run number] [-q] For this workshop, the only arguments to use are: -e, -l, -f, -x and -q. Note that the -f argument, the length of the forecast is specified as: dNN, for example d10 means '10 days'. |
Use the The command will create a create a file ' Submit this job:
Check the job status:
These forecasts complete in about 15mins. |
If you get the error message:
execute the command:
to correct set the environment. |
During the course of the model run, files with names ICM*+* will appear in the experiment directory. These are the model output files. There is 1 per output interval. In these experiments the output interval is every 3 hours.
After the qstatu command shows that the job has completed, there will be a 'batch log file' in the pf001 directory:
oifs_trcray1.o1481792 |
Using your favourite text editor, check this file to make sure the model has run correctly. It's usually best to start from the bottom of the file and scroll up. If you see:
the model has failed. |
The output from each model run goes into a directory named according to this run number. e.g. the default is 'output1'. So job2 would write the output to 'output2' and so on.
If the model fails, there are 2 files to look at in the appropriate 'output' directory:
NODE.001_01 : this is the output from the model as it is running (i.e. all the print/write fortran statements). This is a large file, so in case of errors, start from the bottom up!
oifs.log : this is where the model writes any error messages.
Each model run has a number. This is '1' by default but can be changed using the -x option to the oifs_run command.
oifs_run -e gc1d -l mynamelist -x 2 -f d5 -q |