...
- have learnt about how OpenIFS is installed and organised,
- have run it at T21 resolution and know how to run it serially (1 process) and in parallel with both MPI & OpenMP,
- know how to use grib tools to look at model output.
Connecting to the ECMWF HPC Facilities
In this tutorial, and throughout the entire workshop, you will use one of the Cray XC40 computers which are part of the ECMWF's high-performance computing facility (HPCF).
...
Panel | ||||
---|---|---|---|---|
| ||||
|
Instead of using the login nodes of the HPCF we will use an interactive session to ask for computing resources and fast temporary disk. This also allows to run parallel jobs in the terminal window.Type the following command: The following tasks need to be carried out only once in order to set up the training accounts and the OpenIFS 43r3 model.
Panel | ||||
---|---|---|---|---|
| ||||
|
The changed command line prompt indicates that we have switched from the login node to a pre/post-processing node.
You can close the interactive session by typing exit
which will bring you back to the login node.
On the ECMWF HPCF an interactive session will last for 48 hrs by default, unless the walltime is specified using an additional directive.
OpenIFS directories
In this section we:
...
| |
|
The actions above will ensure that your training account receives required scripts and shell configuration files.
The last command will copy the model binaries and input files to your account for version 43r3.
Instead of using the login nodes of the HPCF we will use an interactive session to ask for computing resources and fast temporary disk. This also allows to run parallel jobs in the terminal window.
Type the following command:
...
Panel | ||||
---|---|---|---|---|
| ||||
|
The oifs-config.ccb.sh
file sets a number of Unix shell environment variables which define the type of OpenIFS compiled installation and location of files.
Make sure you type the source command before you run the model otherwise OpenIFS will not run correctly.
...
| |
|
The changed command line prompt indicates that we have switched from the login node to a pre/post-processing node.
Important: After completing your work you need to close the interactive session by typing
exit
which will bring you back to the login node.
On the ECMWF HPCF an interactive session will last for 48 hrs by default, unless the walltime is specified using an additional directive.
OpenIFS directories
In this section we:
- Set the OpenIFS environment
- Examine the OpenIFS installation
Panel | ||||
---|---|---|---|---|
| ||||
|
The directory contains a number of text files and directories:
Code Block |
---|
% ls
CHANGES CITE COPYING LICENSE NOTICE README READMEs/
bin/ examples/ fcm/ make/ python/ src/ t21test/ |
Directories:
src
- contains all the source code for the model and supporting programs.make
- contains the build configuration files for the FCM compile command. Object files and executables will be in this directory organised according to the type of build (OIFS_BUILD environment variable).t21test
- self contained T21 model run for verifying the installation is correct.ifsdata
- contains additional input files for the model e.g. climatologies. Available as separate tarfiles.
Note:
- OpenIFS has been precompiled on the HPCF.
- All source code has been removed due to licensing restrictions.
- OpenIFS builds 'out-of-source'; this means object (
.o
) files and executables (binary files) are not mixed with the source code. - The
README
file contains information about software requirements, setting up the local compilation environment, and where to get help and support.
OpenIFS T21 test forecasts
In this section of the tutorial, we will run the pre-compiled OpenIFS model on a simple T21 forecast.
You will:
- learn about OpenIFS input and output files.
- learn some switches to control OpenIFS.
- learn how to run the model in parallel.
- learn how to run the acceptability test.
| |
|
The oifs-config.ccb.sh
script sets a number of Unix shell environment variables which define the type of OpenIFS compiled installation and location of files. These settings are specific to version 43r3.
If you do not run this setup script (or alternatively include its content in your Unix shell configuration file) then OpenIFS will not run correctly.
For more information on how to configure OpenIFS, please see the OpenIFS website: http://software.ecmwf.int/oifs/.
Panel | ||||
---|---|---|---|---|
| ||||
|
The directory contains a number of text files and directories:
Code Block |
---|
% ls
bin/ COPYING LICENSE oifs-config.ccb.sh READMEs/
CHANGES examples/ make/ python/ src/
CITE fcm/ NOTICE README t21test/ |
Directories:
src
- contains all the source code for the model and supporting programs.make
- contains the build configuration files for the FCM compile command. Object files and executables will be in this directory organised according to the type of build (OIFS_BUILD environment variable).t21test
- self contained T21 model run for verifying the installation is correct.ifsdata
- contains additional input files for the model e.g. climatologies. Available as separate tarfiles.
Note:
- OpenIFS has been precompiled on the HPCF.
- All source code has been removed due to licensing restrictions.
- OpenIFS builds 'out-of-source'; this means object (
.o
) files and executables (binary files) are not mixed with the source code. - The
README
file contains information about software requirements, setting up the local compilation environment, and where to get help and support.
OpenIFS T21 test forecasts
In this section of the tutorial, we will run the pre-compiled OpenIFS model on a simple T21 forecast.
You will:
- learn about OpenIFS input and output files.
- learn some switches to control OpenIFS.
- learn how to run the model in parallel.
- learn how to run the acceptability test.
Panel | ||||
---|---|---|---|---|
| ||||
Panel | ||||
| ||||
|
The directory t21test
contains a number of files:
Code Block |
---|
% ls t21test ICMGGepc8ICMGGepc8INIT ICMGGepc8INIUA ICMSHepc8INIT ifsdatajob namelists ICMGGepc8INIT ICMSHepc8 ref_021_0072 ICMGGepc8INIUA ifsdata/ README namelists job ref_021_0144run.ppn |
Files beginning with 'ICM'.
These are the input files for this T21 experiment. They are in GRIB format. Do not move them from this directory. OpenIFS expects to find its input files in the same directory as the main executable.
epc8
- this is the Experiment ID. Experiments IDs are used at ECMWF and initial conditions provided by ECMWF will always have an expt id.ICMGGepc8
- 'GG' indicates these contain gridpoint fields.ICMSHepc8
- 'SH' indicates these contains spherical harmonic fields.
job
Simple shell Shell script to run the model. Described in more detail below.
run.ppn
Simple shell script which calls job in an interactive shell environment.
ifsdata
Climate data fields used for T21 test integration. You should not move or rename this directory as the model will expect to find the climate files it needs in a directory of this name.
...
ref_021_0144
This file is reference output for the model tests. The model can be run in 'reference' mode where it checks it is working correctly by comparing some mathematical norms against these files. Reference runs are described in more detail under 'Acceptance testing' below.
Panel | ||||
---|---|---|---|---|
| ||||
Use the |
Panel | ||||
---|---|---|---|---|
| ||||
Run the model:
What happens? |
The model fails. Look in the NODE_001.01
file and find the subroutine traceback. Near the top of the traceback you will find the error messages.
Whenever the model fails, it will produce this traceback (controlled by DR_HOOK=1 in the job file).
Single process test
...
bgColor | #f0f0ff |
---|---|
title | Tasks - Run the model as a single process |
Copy the file namelists
to fort.4
and run the model with a single task and single thread by executing the job script:
Code Block |
---|
% cp namelists fort.4
% ./run.ppn |
norms against these files. Reference runs are described in more detail under 'Acceptance testing' below.
Panel | ||||
---|---|---|---|---|
| ||||
Use the |
Panel | ||||
---|---|---|---|---|
| ||||
Run the model:
What happens? |
The model fails. Look at the standard output (or in the NODE_001.01
file when it is created) and find the subroutine traceback. Near the top of the traceback you will find the error messages.
Whenever the model fails, it will produce this traceback (controlled by DR_HOOK=1 in the job
file).
Single process test
Panel | ||||
---|---|---|---|---|
| ||||
Copy the file
|
The model will expect to find a file called fort.4
in the same directory as the executable. This script copies the executable from make/cce-opt/oifs/bin
.
If the run works you will see output like:
Code Block |
---|
...
17:09:08 STEP 1 H= 0:10 +CPU= 0.276
STEP 1 :## EC_MEMINFO 1 ccbppn01 309 206 0 16530 190 16080 1188 35807 61171 0.2 0.0 s/p
17:09:08 STEP 2 H= 0:20 +CPU= 0.280
STEP 2 :## EC_MEMINFO 1 ccbppn01 309 206 0 16530 190 16011 1188 35740 61170 0.2 0.0 s/p
17:09:08 STEP 3 H= 0:30 +CPU= 0.268
STEP 3 :## EC_MEMINFO 1 ccbppn01 309 206 0 16530 190 16008 1188 35734 61170 0.2 0.0 s/p
17:09:09 STEP 4 H= 0:40 +CPU= 0.264
STEP 4 :## EC_MEMINFO 1 ccbppn01 309 206 0 16530 190 15966 1188 35695 61170 0.2 0.0 s/p
17:09:09 STEP 5 H= 0:50 +CPU= 0.268
STEP 5 :## EC_MEMINFO 1 ccbppn01 309 206 0 16530 190 16008 1188 35734 61171 0.2 0.0 s/p
17:09:09 STEP 6 H= 1:00 |
The model will expect to find a file called fort.4
in the same directory as the executable. This script copies the executable from make/opt/bin
.
If the run works you will see output like:
Code Block |
---|
... signal_drhook(SIGSYS=31): New handler installed at 0x4d06cf; old preserved at 0x0 MPL_BUFFER_METHOD: 2 0 16:03:46 STEP 0 H= 0:00 +CPU= 3.598 16:03:46 STEP 1 H= 0:10 +CPU= 0.535 16:03:47 STEP 2 H= 0:20 +CPU= 0.537004 16:03:48 STEP STEP 6 :## EC_MEMINFO 1 3ccbppn01 H= 0:30 +CPU= 309 0.537 16:03:48206 STEP 4 H= 0 0:40 +CPU= 0.527 16530 16:03:49 STEP 190 5 H=16008 0:50 +CPU= 1188 0.526 16:03:4935734 STEP 61171 6 H= 1:00 +CPU=0.2 0.5300 s/p |
This test runs only 6 a small number of timesteps.
Model output
The model writes its output to a several files.
...
Panel | ||||
---|---|---|---|---|
| ||||
Edit the file Run Do the reported CPU times change? Use the |
...
Code Block |
---|
NUMBER OF THREADS 42 |
to verify the model ran with 4 OpenMP threads.
Panel | ||||
---|---|---|---|---|
| ||||
Edit the file Change the line: Also, edit the fort.4 file and change Rerun the job: Do the reported CPU times change? |
...
Look in the NODE_001.01
output file for the line: "NPROC = 4 2
" to verify that two MPI tasks was used.
...
The model can also be set to use NPROC=4
and 2
and NTHREADS=4
to 2
to use a total of 4 processes. However, this would require a computer with at least 16 4 cores.
Acceptance testing
The final step is to check the model is producing the numerical answers within acceptable limits, even if it runs the short tests above without failing. OpenIFS includes code that will compute internal statistical norms and compare against numbers supplied by ECMWF. The file: ref_021_0144
in the t21test
directory contains statistical norms computed by the model run at ECMWF.
Panel | ||||
---|---|---|---|---|
| ||||
To do the acceptance test, edit the namelists in
change the number of timesteps to 72 to run the model for 12 hours (assuming you have not changed the default timestep of 10mins at T21) and set the variable LREFOUT to TRUE:
|
With LREFOUT=true
, at the last timestep OpenIFS will read the ref_021_0144 file and produce a new file: res_021_0144 (note the similar filenames!). The contents of the file should be similar to:
...
As long as the model reports 'calculations are correct' and the error is less than a few percent then the model is behaving satisfactorily in your compilation and run environment.
How to control model output
...
In this example, the model will write 3 separate output files at the first timestep (0hrs), 3hrs and 9hrs and then no more regardless of how long the model runs for.
How to change the output variables and post-processing
...