...

OpenIFS forecasts are identified by an 'experiment id', a four letter string. An experiment could consist of a single forecast date or multiple starting dates. It can be of any length and could also include a restarted forecast. However, an experiment only has one horizontal and vertical resolution.

Initial files

The initial files containing 2D and 3D fields to start the forecast are contained in file that beging with ICM* : ICMGG* are the initial gridpoint files, ICMSH* are the initial spectral fields. ICMCL* is the file containing the climatological forcing fields.

The experiment id is contained the names of the initial files, and it is encoded into the GRIB messages within those files.

Listing experiment id

The command 'exptid' can be used to check and change the experiment id in the GRIB files.

troifs0@cca-login3:> cd inidir/ob01/2015110100
troifs0@cca-login3:> exptid ICMSHob00INIT
In file ICMSHob00INIT, values of key experimentVersionNumber are:
ob00

The 'experimentVersionNumber' is the GRIB parameter encoded in the GRIB fields.

This can also be seen using the 'grib_ls' command to list the contents of the file:

Use the 'grib_ls' command to examine the initial files:

grib_ls ICMGGob00INIT
grib_ls ICMSHob00INIT

OpenIFS namelist

The file: fort.4 is the model 'NAMELIST'. It contains a list of variable settings or 'switches' that control what the model does. These variables are grouped into separate fortran namelists.

The model will read this file when it starts up. There are many options to control the model. For more information, it is best and recommended to check the comments in the code.

The wave model, WAM, also has a separate namelist. For this workshop, none of the switches in this file needs to be changed.

Listing the experiment id

The command 'exptid' can be used to check and change the experiment id in the GRIB files.

troifs0@cca-login3:> cd inidir/ob01/2015110100
troifs0@cca-login3:> exptid ICMSHob00INIT
In file ICMSHob00INIT, values of key experimentVersionNumber are:
ob00

The 'experimentVersionNumber' is the GRIB parameter encoded in the GRIB fields.

This can also be seen using the 'grib_ls' command to list the contents of the file:

troifs0@cca-login3:> grib_ls -p shortName,typeOfLevel,dataDate,experimentVersionNumber ICMSHob00INIT
ICMSHob00INIT
shortName

troifs0@cca-login3:> grib_ls -p shortName,typeOfLevel,dataDate,experimentVersionNumber ICMSHob00INIT
ICMSHob00INIT
shortName                typeOfLevel              dataDate         typeOfLevel              dataDate                 experimentVersionNumber  
t                        hybrid                   20151101                 ob00                    
t                        hybrid                   20151101                 ob00                    
t                        hybrid                   20151101                 ob00  
.......    

...

To setup the experiment directory structure, use the 'createEX' command. To see what arguments it takes, use the command:

createEX -h

The most basic form of the command is:

...

troifs0@ccb-login3:> ls -l 00
total 16
lrwxrwxrwx 1 troifs0 ectrain   52 May 25 18:15 255l_2 -> /fwsm/lb/project/openifs/ifsdata/40r1/climate/255l_2
lrwxrwxrwx 1 troifs0 ectrain   16 May 25 18:15 ICMCLob50INIT -> ../ICMCLob50INIT
lrwxrwxrwx 1 troifs0 ectrain   16 May 25 18:15 ICMGGob50INIT -> ../ICMGGob50INIT
lrwxrwxrwx 1 troifs0 ectrain   17 May 25 18:15 ICMGGob50INIUA -> ../ICMGGob50INIUA
lrwxrwxrwx 1 troifs0 ectrain   16 May 25 18:15 ICMSHob50INIT -> ../ICMSHob50INIT
lrwxrwxrwx 1 troifs0 ectrain   11 May 25 18:15 cdwavein -> ../cdwavein
-rw-r----- 1 troifs0 ectrain 9886 May 25 18:15 fort.4
lrwxrwxrwx 1 troifs0 ectrain   49 May 25 18:15 ifsdata -> /fwsm/lb/project/openifs/ifsdata/40r1/climatology
lrwxrwxrwx 1 troifs0 ectrain   40 May 25 18:15 rtables -> /fwsm/lb/project/openifs/ifsdata/rtables
lrwxrwxrwx 1 troifs0 ectrain   12 May 25 18:15 sfcwindin -> ../sfcwindin
lrwxrwxrwx 1 troifs0 ectrain   13 May 25 18:15 specwavein -> ../specwavein
lrwxrwxrwx 1 troifs0 ectrain   10 May 25 18:15 uwavein -> ../uwavein
lrwxrwxrwx 1 troifs0 ectrain   18 May 25 18:15 wam_grid_tables -> ../wam_grid_tables
-rw-r----- 1 troifs0 ectrain 2220 May 25 18:15 wam_namelist
lrwxrwxrwx 1 troifs0 ectrain   16 May 25 18:15 wam_subgrid_0 -> ../wam_subgrid_0
lrwxrwxrwx 1 troifs0 ectrain   16 May 25 18:15 wam_subgrid_1 -> ../wam_subgrid_1
lrwxrwxrwx 1 troifs0 ectrain    16 May 25 18:15 wam_subgrid_2 -> ../wam_subgrid_2

Notice the links back to the parent directory. To save space, the files that do not change between members are kept in the parent directory.

Only the model namelists, fort.4 for the atmosphere, wam_namelist for the wave model, are individual to a forecast ensemble member directory.

Running the forecast experiment

blah blah blah

Postprocessing

To postprocess the OpenIFS output files,

Useful Unix commands

16 May 25 18:15 wam_subgrid_2 -> ../wam_subgrid_2

Notice the links back to the parent directory. To save space, the files that do not change between members are kept in the parent directory.

Only the model namelists, fort.4 for the atmosphere, wam_namelist for the wave model, are individual to a forecast ensemble member directory.

Running the forecast experiment

The command 'oifs_run' is normally used to create and submit the batch job to run the model. This command has a large number of options. For this workshop, this command has been configured with the correct defaults for the T255 resolution seasonal length forecasts.

However, it would need to be run in each forecast member directory, which would be tedious.

The command 'run_all_ens' has been created to speed up submitting the batch jobs to run all the forecast members with one command.

For our 'ob01' experiment, this command would create all the jobs, correctly configure the namelist and submit the job. There will be 5 jobs, one forecast for each ensemble member:

troifs0@cca-login3:~> run_all_ens -e ob01 -d scratch/ob01 -q

The -q option here ensures the batch jobs are submitted to the batch queue. If -q is not specified then the batch script 'job' is only created in each ensemble directory and not submitted.

This will produce lengthy output like this:

.........
Ensemble member: 05
60c60
<       NENSFNB=5,
---
>       NENSFNB=0,                 ! Ensemble forecast number
Running command: oifs_run -e ob01 -x 1 -q
Copied /home/ectrain/troifs0/bin/cce-opt/master.exe to current directory
Using existing namelist in : fort.4
OpenIFS job created: job1
Submitted job: 8430146.ccapar
......

Setting the ensemble member

The command 'run_all_ens' does this step, this section provides further information.

In IFS, each ensemble member uses the stochastic physics schemes to generate uncertainty. A random number 'seed' is used by the stochastic scheme to generate a different forecast.

This random number seed is changed by altering the ensemble member value, NENSFNB, in the model's namelist, fort.4. Each ensemble member must have a unique number and therefore random number seed, in order to produce a different forecast.

The random seed is also date dependent. This means that for the same date and same ensemble member, the forecast is reproducible. But for the same ensemble member starting from a different date/time, the random seed will be different.

CTYPE="pf",        ! the type of forecast: 'pf=perturbed', 'cf=control'. The control (unperturbed) forecast is only used in medium-range forecasts, not in seasonal forecasts.
NENSFNB=2,         ! the ensemble member number. 
LSTOPH_SPBS=true,  ! enables the stochastic backscatter scheme in the model dynamics. This is only used in medium-range forecasts and will be 'false' for seasonal forecasts.
LSPSDT=true,       ! enables the stochastic scheme for the physics tendencies.

The only namelist variable that needs changing in these experiments is NENSFNB, which the run_all_ens command does for you.

Create and submit Cray batch job

To create the batch job, use the oifs_run command. This creates a small batch job file ready to submit.

The run_all_ens command runs this command for each ensemble member, but you may need to use the oifs_run command if the batch job script 'job' in each ensemble directory needs to be recreated. Note if you need to rerun the experiment, with no changes, simply resubmit the job: qsub job, rather than use oifs_run to recreate the job.

Checking the model output

During the course of the model run, files with names like ICMGGob01+00000, ICMGGob01+00012 etc will appear in the ensemble experiment directory.

These are the model output files. There is 1 per output interval. In these experiments the output interval is every 12 hours. It can be changed in the namelist, fort.4 but it's strongly recommended for these workshop, to keep the output interval at 12 hrs as this saves storage space.

After the qstatu command shows that the job has completed, there will be a 'batch log file' in the ensemble directory:

oifs_troifs1.o1481792

Using your favourite text editor, check this file to make sure the model has run correctly.  It's usually best to start from the bottom of the file and scroll up.

If you see:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   OPENIFS JOB FAILED
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

the model run has failed.

The output from each model run goes into a directory named 'output1'.

If the model fails, there are 2 files to look at in the 'output' directory:

NODE.001_01 : this is the output from the model as it is running (i.e. all the print/write fortran statements). This is a large file, so in case of errors, start from the bottom up!

oifs.log : this is where the model writes any error messages.

Postprocessing: preparing files for Metview

NEEDS UPDATING!!!!

The next step is to take the output of the successful run and prepare it for the Metview based exercises.

As the data files and Metview scripts have been optimized by selecting specific areas, we have made a script available to do the work.

 oifs_to_mv -e 00

This will generate files in scratch/metview.

If errors are encountered please let us know. If the command completes successfully (it will take a while), this task is complete. There is no need to inspect the files.

Once the script has finished, the results files can be transferred to ICTP to be used on the classroom PC with metview.

Useful Unix commands

qsub job

qstatu

rm -rf 2015110100
rm -rf 2015110[1-5]00

...

Quota for $HOME and $PERM:
Disk quotas for user troifs0 (uid 16144): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
cnasa1:/vol/home
                   226M    480M    500M             132   20000   22000        
Disk quotas for user troifs0 (uid 16144): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
cnasa2:/vol/perm
                      0  26624M  27648M               1    200k    210k        

Quota for $SCRATCH ($TEMP) including $SCRATCHDIR ($TMPDIR):
Disk quotas for user troifs0 (uid 16144):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
  /lus/snx11062 767099824  32212254720 32212254720       -  116570  5000000 5000000       -
Disk quotas for group ectrain (gid 1400):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
  /lus/snx11062 767105640       0       0       -  118078       0       0       -