| Section | ||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...
Each group or participant will have a training user account on the ECMWF system, beginning with 'troifs'. This is different from the user account on the classroom computers.
First login to the ECMWF gateway computer. From here you will login to the HPCF.
| Code Block | ||
|---|---|---|
| ||
ssh troifsXX@ecgb06.ecmwf.int # substitute your user id for XX troifsXX@ecgb06.ecmwf.int's password: # give your password when prompted using the securID hardware token (one time password) |
Next login to the Cray login nodes:
| Code Block | ||
|---|---|---|
| ||
ssh cca troifs@cca-login2:~> |
...
This section will explain how to copy and change the experiment id.
Initial Initial files for the observed SST experiment, 'ob00', and the climatological SST experiment 'clim', have been created and are made available in the directory : /perm/rd/openifs/oifs_workshop_2017/expts-inidata/
...
ob00/2015110100 to ob00/2015110100 etc.
...
You do not need to run these experiments, the forecasts from a 10 member ensemble are made available on the classroom computers.
Copy previous experiment initial files
Before creating the experiment, we first need to create the initial data. If you plan to rerun the ob00 or clim experiments and use the initial data 'as-is' you can skip this step.
...
First step is to make a copy of the ob00 data to your /scratch directory:
| Code Block | ||||
|---|---|---|---|---|
| ||||
cd scratch # starting from your home directory mkdir -p inidir/ob01 # our new experiment id cd inidir/ob01 cp -rL /perm/rd/openifs/oifs_workshop_2017/expts-inidata/ob00/2015110100 . # those last characters are a 'space' and then a 'fullstop', '.' means "here". |
...
Ignore any errors about files not being copies because of permission problems.
| Code Block | ||||
|---|---|---|---|---|
| ||||
cd inidir/ob01 cp -rL /perm/rd/openifs/oifs_workshop_2017/expts-inidata/ob00/2015110[1-5]* . |
Please only copy the initial dates you intend to use. Each date uses 0.5 Gbyte of file storage.
| Code Block | ||||
|---|---|---|---|---|
| ||||
cd inidir/ob01 cp -rL /perm/rd/openifs/oifs_workshop_2017/expts-inidata/ob00/201511* . # this will be slow! |
...
Use the 'grib_ls' command to examine the initial files:
| Code Block | ||
|---|---|---|
| ||
grib_ls ICMGGob00INIT grib_ls ICMSHob00INIT |
...
The command 'exptid' can be used to check and change the experiment id in the GRIB files.
| Code Block | ||||
|---|---|---|---|---|
| ||||
troifs0@cca-login3:> cd inidir/ob01/2015110100 troifs0@cca-login3:> exptid ICMSHob00INIT In file ICMSHob00INIT, values of key experimentVersionNumber are: ob00 |
...
This can also be seen using the 'grib_ls' command to list the contents of the file:
| Code Block | ||
|---|---|---|
| ||
troifs0@cca-login3:> grib_ls -p shortName,typeOfLevel,dataDate,experimentVersionNumber ICMSHob00INIT ICMSHob00INIT shortName typeOfLevel dataDate experimentVersionNumber t hybrid 20151101 ob00 t hybrid 20151101 ob00 t hybrid 20151101 ob00 ....... |
The 'exptid' and 'grib_ls' commands can also be used for multiple files:
| Code Block | ||
|---|---|---|
| ||
troifs0@cca-login3:> exptid ICM* In file ICMCLob00INIT, values of key experimentVersionNumber are: 0001 In file ICMGGob00INIT, values of key experimentVersionNumber are: 0001 ob00 In file ICMGGob00INIUA, values of key experimentVersionNumber are: ob00 In file ICMSHob00INIT, values of key experimentVersionNumber are: ob00 |
...
Use the exptid command to set the new experiment id to 'ob01':
| Code Block | ||
|---|---|---|
| ||
troifs0@cca-login3:> exptid -n ob01 ICM*ob00* Changing expid from 'ob00' to 'ob01' for ICMCLob00INIT and writing to new file ICMCLob01INIT. Changing expid from 'ob00' to 'ob01' for ICMGGob00INIT and writing to new file ICMGGob01INIT. Changing expid from 'ob00' to 'ob01' for ICMGGob00INIUA and writing to new file ICMGGob01INIUA. Changing expid from 'ob00' to 'ob01' for ICMSHob00INIT and writing to new file ICMSHob01INIT. |
Verify the command worked by checking the experimentVersionNumber in the new files:
| Code Block | ||
|---|---|---|
| ||
troifs0@cca-login3:> exptid *ob01* In file ICMCLob01INIT, values of key experimentVersionNumber are: 0001 In file ICMGGob01INIT, values of key experimentVersionNumber are: 0001 ob01 In file ICMGGob01INIUA, values of key experimentVersionNumber are: ob01 In file ICMSHob01INIT, values of key experimentVersionNumber are: ob01 |
To save space, you can delete the ob00 files (these will always be available in the directory where they were copied from)
| Code Block | ||
|---|---|---|
| ||
troifs0@cca-login3:> rm ICM*ob00* rm: remove regular file `ICMCLob00INIT'? y rm: remove regular file `ICMGGob00INIT'? y rm: remove regular file `ICMGGob00INIUA'? y rm: remove regular file `ICMSHob00INIT'? y |
...
To setup the experiment directory structure, use the 'createEX' command. To see what arguments it takes, use the command:
| Code Block | ||||
|---|---|---|---|---|
| ||||
createEX -h |
The most basic form of the command is:
| Code Block | ||
|---|---|---|
| ||
createEX --date 2015110100 -e ob00 |
...
Following the creation of the initial files for the experiment id 'ob01' given above, let's create a forecast experiment with 5 ensemble members for a single forecast date:
| Code Block | ||
|---|---|---|
| ||
troifs0@cca-login3:~> createEX -d 2015110100 -e ob01 -i scratch/inidir -m 5 |
which generates the output:
| Code Block | ||
|---|---|---|
| ||
Creating directory structure for experiment ob01 in directory /scratch/ectrain/troifs0/ob01/... Date : 2015110100 Copying files from directory (inidir): scratch/inidir Date: 2015110100 Created forecast experiment directory : /scratch/ectrain/troifs0/ob01/2015110100/00/ Linking files and copying namelist. Using IFS data directory: /fwsm/lb/project/openifs/ifsdata/ Created forecast experiment directory : /scratch/ectrain/troifs0/ob01/2015110100/01/ Linking files and copying namelist. Using IFS data directory: /fwsm/lb/project/openifs/ifsdata/ Created forecast experiment directory : /scratch/ectrain/troifs0/ob01/2015110100/02/ Linking files and copying namelist. Using IFS data directory: /fwsm/lb/project/openifs/ifsdata/ Created forecast experiment directory : /scratch/ectrain/troifs0/ob01/2015110100/03/ Linking files and copying namelist. Using IFS data directory: /fwsm/lb/project/openifs/ifsdata/ Created forecast experiment directory : /scratch/ectrain/troifs0/ob01/2015110100/04/ Linking files and copying namelist. Using IFS data directory: /fwsm/lb/project/openifs/ifsdata/ Created forecast experiment directory : /scratch/ectrain/troifs0/ob01/2015110100/05/ Linking files and copying namelist. Using IFS data directory: /fwsm/lb/project/openifs/ifsdata/ All done: /scratch/ectrain/troifs0/ob01/ ready. |
...
If we look at in the directory: /scratch/ectrain/troifs0/ob01/2015110100, we see:
| Code Block | ||
|---|---|---|
| ||
troifs0@ccb-login3:> cd scratch/ob01/2015110100 troifs0@ccb-login3:> ls 00 03 ICMCLob50INIT ICMSHob50INIT specwavein wam_subgrid_0 01 04 ICMGGob50INIT cdwavein uwavein wam_subgrid_1 02 05 ICMGGob50INIUA sfcwindin wam_grid_tables wam_subgrid_2 |
Each member will be run in the numbered directories: "00", "01", "02", "03", "04", and "05". These contain:
| Code Block | ||
|---|---|---|
| ||
troifs0@ccb-login3:> ls -l 00 total 16 lrwxrwxrwx 1 troifs0 ectrain 52 May 25 18:15 255l_2 -> /fwsm/lb/project/openifs/ifsdata/40r1/climate/255l_2 lrwxrwxrwx 1 troifs0 ectrain 16 May 25 18:15 ICMCLob50INIT -> ../ICMCLob50INIT lrwxrwxrwx 1 troifs0 ectrain 16 May 25 18:15 ICMGGob50INIT -> ../ICMGGob50INIT lrwxrwxrwx 1 troifs0 ectrain 17 May 25 18:15 ICMGGob50INIUA -> ../ICMGGob50INIUA lrwxrwxrwx 1 troifs0 ectrain 16 May 25 18:15 ICMSHob50INIT -> ../ICMSHob50INIT lrwxrwxrwx 1 troifs0 ectrain 11 May 25 18:15 cdwavein -> ../cdwavein -rw-r----- 1 troifs0 ectrain 9886 May 25 18:15 fort.4 lrwxrwxrwx 1 troifs0 ectrain 49 May 25 18:15 ifsdata -> /fwsm/lb/project/openifs/ifsdata/40r1/climatology lrwxrwxrwx 1 troifs0 ectrain 40 May 25 18:15 rtables -> /fwsm/lb/project/openifs/ifsdata/rtables lrwxrwxrwx 1 troifs0 ectrain 12 May 25 18:15 sfcwindin -> ../sfcwindin lrwxrwxrwx 1 troifs0 ectrain 13 May 25 18:15 specwavein -> ../specwavein lrwxrwxrwx 1 troifs0 ectrain 10 May 25 18:15 uwavein -> ../uwavein lrwxrwxrwx 1 troifs0 ectrain 18 May 25 18:15 wam_grid_tables -> ../wam_grid_tables -rw-r----- 1 troifs0 ectrain 2220 May 25 18:15 wam_namelist lrwxrwxrwx 1 troifs0 ectrain 16 May 25 18:15 wam_subgrid_0 -> ../wam_subgrid_0 lrwxrwxrwx 1 troifs0 ectrain 16 May 25 18:15 wam_subgrid_1 -> ../wam_subgrid_1 lrwxrwxrwx 1 troifs0 ectrain 16 May 25 18:15 wam_subgrid_2 -> ../wam_subgrid_2 |
...
For our 'ob01' experiment, this command would create all the jobs, correctly configure the namelist and submit the job. There will be 5 jobs, one forecast for each ensemble member:
| Code Block | ||||
|---|---|---|---|---|
| ||||
troifs0@cca-login3:~> run_all_ens -e ob01 -d scratch/ob01 -q |
...
This will produce lengthy output like this:
| Code Block | ||
|---|---|---|
| ||
......... Ensemble member: 05 60c60 < NENSFNB=5, --- > NENSFNB=0, ! Ensemble forecast number Running command: oifs_run -e ob01 -x 1 -q Copied /home/ectrain/troifs0/bin/cce-opt/master.exe to current directory Using existing namelist in : fort.4 OpenIFS job created: job1 Submitted job: 8430146.ccapar ...... |
...
The random seed is also date dependent. This means that for the same date and same ensemble member, the forecast is reproducible. But for the same ensemble member starting from a different date/time, the random seed will be different.
| Panelcode | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
CTYPE="pf", ! the type of forecast: 'pf=perturbed', 'cf=control'. The control (unperturbed) forecast is only used in medium-range forecasts, not in seasonal forecasts. NENSFNB=2, ! the ensemble member number. LSTOPH_SPBS=true, ! enables the stochastic backscatter scheme in the model dynamics. This is only used in medium-range forecasts and will be 'false' for seasonal forecasts. LSPSDT=true, ! enables the stochastic scheme for the physics tendencies. |
The only namelist variable that needs changing in these experiments is NENSFNB, which the run_all_ens command does for you.
Create and submit Cray batch job
...
During the course of the model run, files with names like ICMGGob01+00000, ICMGGob01+00012 etc will appear in the ensemble experiment member directory.
These are the model output files. There is 1 per output intervalstep. In these experiments the output interval is every 12 hours. It can be changed in the namelist, fort.4 but it's strongly recommended for these workshop, to keep the output interval at 12 hrs as this saves storage space.
...
Using your favourite text editor, check this file to make sure the model has run correctly. It's usually best to start from the bottom of the file and scroll up.
If you see:
| Code Block | ||
|---|---|---|
| ||
!!!!!!!!!!!!!!!!!!!!!!!!!!!!! OPENIFS JOB FAILED !!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
...
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
When the model completes the forecast successfully, the following files will be found in the output directory. Files with '+' in the name are the model output files. ICMSH are the spectral fields, ICMGG are the gridpoint fields. ICMGGoifs+000000 ICMGGoifs+000048 ICMGGoifs+000096 ICMGGoifs+000144 ICMSHoifs+000016 ICMSHoifs+000064 ICMSHoifs+000112 ICMSHoifs+000160 oifs.log ICMGGoifs+000016 ICMGGoifs+000064 ICMGGoifs+000112 ICMGGoifs+000160 ICMSHoifs+000032 ICMSHoifs+000080 ICMSHoifs+000128 NODE.001_01 ICMGGoifs+000032 ICMGGoifs+000080 ICMGGoifs+000128 ICMSHoifs+000000 ICMSHoifs+000048 ICMSHoifs+000096 ICMSHoifs+000144 ifs.stat ICMGGclim_10u ICMGGclim_ci ICMGGclim_lsp ICMGGclim_sd ICMGGclim_sst ICMGGclim_stl4 ICMGGclim_swvl3 ICMGGclim_tsr ICMSHclim_sp NODE.001_01 ICMGGclim_10v ICMGGclim_cp ICMGGclim_msl ICMGGclim_slhf ICMGGclim_stl1 ICMGGclim_str ICMGGclim_swvl4 ICMGGclim_ttr ICMSHclim_t ifs.stat ICMGGclim_2d ICMGGclim_e ICMGGclim_nsss ICMGGclim_sshf ICMGGclim_stl2 ICMGGclim_swvl1 ICMGGclim_tcc ICMSHclim_d ICMSHclim_vo oifs.log ICMGGclim_2t ICMGGclim_ewss ICMGGclim_q ICMGGclim_ssr ICMGGclim_stl3 ICMGGclim_swvl2 ICMGGclim_tp ICMSHclim_lnsp ICMSHclim_z |
Postprocessing: preparing files for Metview
NEEDS UPDATING!!!!
The next step is to take the output of the successful run and prepare it for the Metview based exercises.
...
| Code Block | ||
|---|---|---|
| ||
quota |
whicih which will produce output similar to the following. The '$SCRATCH' quota is important as this is the total limit for the filesystem /scratch where your experiments are stored.
...