Versions Compared

Old Version 27

changes.mady.by.user Unknown User (nagc)

Saved on

compared with

New Version Current

changes.mady.by.user Unknown User (nagc)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Section


Column


 

Panel
borderColorblue
bgColor#e6ffff
borderWidth1
borderStylesolid

This tutorial explains how to create a new experiment from an existing one by copying the initial files and setting up the experiment directory.

ECMWF HPCF

ECMWF operates a Cray based High Performance Computing Facility (HPCF). It consists of two identical Cray XC40 clusters with their own storage, but with equal access to the high performance working storage of the other cluster. This provides the benefit of having one very large system but the dual clusters add significantly to the resiliency of the system.

For more details about the ECMWF HPCF, please see: http://www.ecmwf.int/en/computing/our-facilities/supercomputer

All the OpenIFS experiments will make use of the ECMWF HPCF. During the workshop you will login to the 'front-end' machines and submit 'batch job's to the HPCF.

The scripts described below are provided to make preparing and submitting ensemble batch jobs easier. Useful commands are listed at the end of this tutorial.


Column
width40%


Panel
borderStyledotted
titleOn this page...

Table of Contents



...

Code Block
languagebash
ssh troifsXX@ecgb06troifsXX@ecaccess.ecmwf.int		               # substitute your user id for XX
troifsXX@ecgb06troifsXX@ecaccess.ecmwf.int's password:              # give your password when prompted using the securID hardware token (one time password)

Next login to the Cray login nodesWhen prompted select 'cca' :

Code Block
languagebash
ssh cca
troifs@cca-login2:~> 
troifs1@140.105.20.128's password: 
Select hostname (ecgate, cca, ccb) [ecgate]: cca

'cca' is the "front-end" computer to the ECMWF Cray high performance computer facility.

The contents of the account should look like:

...

Code Block
languagebash
titleTo copy multiple dates. Example 1: first 5 days
cd inidir/ob01
cp -rL /perm/rd/openifs/oifs_workshop_2017/expts-inidata/ob00/2015110[1-5]* .     # this will take some time

Please Please only copy the initial dates you intend to use. Each date uses 0.5 Gbyte of file storage.

Code Block
languagebash
titleTo copy multiple dates. Example 2: copy first 3 dates
cd inidir/ob01
cp -rL /perm/rd/openifs/oifs_workshop_2017/expts-inidata/ob00/2015110[1-3]* .      # this will take some time.

Change experiment id

OpenIFS forecasts are identified by an 'experiment id', a four letter string.

Change experiment id

OpenIFS forecasts are identified by an 'experiment id', a four letter string.

An experiment could consist of a single forecast date or multiple starting dates. It can be of any length and could also include a restarted forecast. However, An experiment could consist of a single forecast date or multiple starting dates. It can be of any length and could also include a restarted forecast. However, an experiment only has one horizontal and vertical resolution.

...

Code Block
languagebash
troifs0@cca-login3:~> createEX -d 2015110100 -e ob01 -i scratch/inidir -m 910

We tell this command it can find our new initial files in scratch/inidir. By default, this command will create the experiment directory also in scratch. Be careful not to confuse the initial data directory with the experiment directory. These should be kept separate.

Info

Note, one ensemble member '00' (or a ' control') is always created, so the -m argument gives lists the total number of additional ensemble forecastsmembers and defaults to 1. Directories are numbered starting from zero.

Only use 10 members in total otherwise you will not be able to compare with the climatological SST experiment which uses 10 members.

...

Code Block
languagebash
titleKey ensemble namelist variables
CTYPE="pf",        ! the type of forecast: 'pf=perturbed', 'cf=control'. The control (unperturbed) forecast is only used in medium-range forecasts, not in seasonal forecasts.
NENSFNB=2,         ! the ensemble member number. 
LSTOPH_SPBS=true,  ! enables the stochastic backscatter scheme in the model dynamics. This is only used in medium-range forecasts and will be 'false' for seasonal forecasts.
LSPSDT=true,       ! enables the stochastic scheme for the physics tendencies.

...

tendencies.

The only namelist variable that needs changing in these experiments is NENSFNB, which the run_all_ens command does for you.

Note

The stochastic backscatter scheme LSTOPH_SPBS should be disabled for OpenIFS 43r3 and beyond as it's use has been deprecated since 40r1. It's impact is very small, less so on the newer cubic grids, and is not recommended.


Create and submit Cray batch job

...

Panel
bgColorwhite
titleBGColorwhite
titleContents of output directory after successful run

When the model completes the forecast successfully, the following files will be found in the output directory. ICMSH are the spectral fields, ICMGG are the gridpoint fields.

ICMGGclim_10u  ICMGGclim_ci    ICMGGclim_lsp   ICMGGclim_sd    ICMGGclim_sst   ICMGGclim_stl4   ICMGGclim_swvl3  ICMGGclim_tsr   ICMSHclim_sp  NODE.001_01

ICMGGclim_10v  ICMGGclim_cp    ICMGGclim_msl   ICMGGclim_slhf  ICMGGclim_stl1  ICMGGclim_str    ICMGGclim_swvl4  ICMGGclim_ttr   ICMSHclim_t   ifs.stat

ICMGGclim_2d   ICMGGclim_e     ICMGGclim_nsss  ICMGGclim_sshf  ICMGGclim_stl2  ICMGGclim_swvl1  ICMGGclim_tcc    ICMSHclim_d     ICMSHclim_vo  oifs.log

ICMGGclim_2t   ICMGGclim_ewss  ICMGGclim_q     ICMGGclim_ssr   ICMGGclim_stl3  ICMGGclim_swvl2  ICMGGclim_tp     ICMSHclim_lnsp  ICMSHclim_z

ICMGGclim_2d   ICMGGclim_e     ICMGGclim_nsss  ICMGGclim_sshf  ICMGGclim_stl2  ICMGGclim_swvl1  ICMGGclim_tcc    ICMSHclim_d     ICMSHclim_vo  oifs.log

ICMGGclim_2t   ICMGGclim_ewss  ICMGGclim_q     ICMGGclim_ssr   ICMGGclim_stl3  ICMGGclim_swvl2  ICMGGclim_tp     ICMSHclim_lnsp  ICMSHclim_z

Rerun the model job

A small file 'job1' (or jobN where N is the run number you used), is created in each of the ensemble member directories; 00, 01, 02, etc.

If for any reason the model fails, once you have determined the problem and corrected it, then the run can be submitted by:

Code Block
qsub job1

There is no need to rerun the run_all_ens command as this will resubmit ALL the ensemble members again.

Anchor
postprocessing
postprocessing
Postprocessing: preparing monthly means for plotting with Metview

...

This command only processes one date at a time. It needs to be run run separately for multiple dates in the same experiment id.

Monthly mean files

After the oifs_to_mv command finishes, the monthly mean files can be found in the directory 'mmeans' inside the top level date directory. For example, for experiment id ob01, date 2015110100, the directory is: ob01/2015110100/mmeans.

...

The monthly mean files can be transferred to ICTP with the ftp sftp command, to be used on the classroom PC with metview.

Useful Unix commands

 


Code Block
titleSubmit a job to the Cray..
qsub job

...

Code Block
Quota for $HOME and $PERM:
Disk quotas for user troifs0 (uid 16144): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
cnasa1:/vol/home
                   226M    480M    500M             132   20000   22000        
Disk quotas for user troifs0 (uid 16144): 
     Filesystem  blocks   quota   limit   grace   files   quota   limit   grace
cnasa2:/vol/perm
                      0  26624M  27648M               1    200k    210k        

Quota for $SCRATCH ($TEMP) including $SCRATCHDIR ($TMPDIR):
Disk quotas for user troifs0 (uid 16144):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
  /lus/snx11062 767099824  32212254720 32212254720       -  116570  5000000 5000000       -
Disk quotas for group ectrain (gid 1400):
     Filesystem  kbytes   quota   limit   grace   files   quota   limit   grace
  /lus/snx11062 767105640       0       0       -  118078       0       0       -

...


 Excerpt Include
Credits
Credits
nopaneltrue