Page History

...

Introduction

OpenIFS is an ECMWF led project which provides an easy-to-use, exportable version of the IFS system in use at ECMWF for operational weather forecasting. The project aims to develop and promote research, teaching and training on numerical weather prediction (NWP) and NWP-related topics with academic and research institutions.

In this tutorial, OpenIFS has been pre-installed and pre-compiled for you. If you want to learn more about how OpenIFS can be downloaded, configured and installed, please see the OpenIFS website at: http://software.ecmwf.int/oifs/.

In this tutorial...

What you will do:

Connect to the ECMWF HPC Facility
Explore installed OpenIFS files
Run T21 model forecast
Verify the model works in serial (1 process) and parallel (2 tasks & 2 threads)
Carry out acceptance test
Use the model namelist to change settings
Explore model output data (GRIB)

At the end of this tutorial, you will:

have learnt about how OpenIFS is installed and organised,
have run it at T21 resolution and know how to run it serially (1 process) and in parallel with both MPI & OpenMP,
know how to use grib tools to look at model output.

Connecting to the ECMWF HPC Facilities

In this tutorial, and throughout the entire workshop, you will use a one of the Cray XC40 computers which is are part of the ECMWF's high-performance computing facility (HPCF).

...

Panel

bgColor	#f0f0ff
title	Tasks - Connect to HPCF

Start the Mobaxterm application and open a local terminal
Login to the RACC using ssh -X cluster.act.rdg.ac.uk
On the cluster type
Open a terminal window by clicking on the 'terminal' icon in the toolbar.
Type the command: ssh troifsXX@ecaccesstroifsX@ecaccess.ecmwf.int Note: Instead of troifsXX troifsX you should use your ECMWF training user-ID
You will be prompted for the hostname with a choice between ecgate, cca and ccb. You should select ccb.
When you have completed your work you can disconnect from ccb by by typing exit at the command prompt.

Instead of using the login nodes of the HPCF we will use an interactive session to ask for computing resources and fast temporary disk. This also allows to run parallel jobs in the terminal window.Type the following command: The following tasks need to be carried out only once in order to set up the training accounts and the OpenIFS 43r3 model.

Panel

bgColor	#f0f0ff
title	Tasks - Start interactive session

% qsub -I -q nf -l EC_total_tasks=12 -l EC_job_tmpdir=10G -l EC_memory_per_task=2G
qsub: waiting for job 7215630.ccbpar to start
qsub: job 7215630.ccbpar ready

The changed command line prompt indicates that we have switched from the login node to a post-processing node.

You can close the interactive session by typing exit which will bring you back to the login node.

On the ECMWF HPCF an interactive session will last for 48 hrs by default, unless the walltime is specified using an additional directive.

OpenIFS directories

In this section we:

...

Set up training account - do this only once after the first login

After the first login type the following command: /home/ectrain/troifs0/setup-cray/my43
Log out of the training account by typing exit
Login once more using ssh troifsX@ecaccess.ecmwf.int
Type the following command: get43

The actions above will ensure that your training account receives required scripts and shell configuration files.

The last command will copy the model binaries and input files to your account for version 43r3.

Instead of using the login nodes of the HPCF we will use an interactive session to ask for computing resources and fast temporary disk. This also allows to run parallel jobs in the terminal window.

Type the following command:

...

Panel

bgColor	#f0f0ff
title	Tasks - Set OpenIFS environment

Connect to ccb as above.
Type the command: cd openifs.
Type the command: source ./oifs.setup

The 'oifs.setup' file sets a number of Unix shell environment variables which define the type of OpenIFS compiled installation and location of files.

Make sure you type the source command every time you start the VM otherwise OpenIFS will not run correctly.

...

Start interactive session

% qsub -I -q df -l EC_total_tasks=6 -l EC_job_tmpdir=10G -l EC_memory_per_task=2G
qsub: waiting for job 7215630.ccbpar to start
qsub: job 7215630.ccbpar ready

The changed command line prompt indicates that we have switched from the login node to a pre/post-processing node.

Important: After completing your work you need to close the interactive session by typing exit which will bring you back to the login node.

On the ECMWF HPCF an interactive session will last for 48 hrs by default, unless the walltime is specified using an additional directive.

OpenIFS directories

In this section we:

Set the OpenIFS environment
Examine the OpenIFS installation

Type: cd openifs

Type: ls

Panel

bgColor	#f0f0ff
title	Tasks - Examine directory layout

Code Block
ls

Directories:

src - contains all the source code for the model and supporting programs.
make - contains the build configuration files for the FCM compile command. Object files and executables will be in this directory organised according to the type of build (OIFS_BUILD environment variable).
t21test - self contained T21 model run for verifying the installation is correct.
ifsdata - contains additional input files for the model e.g. climatologies. Available as separate tarfiles.

Note:

OpenIFS has been precompiled on the virtual machine.
All source code has been removed due to licensing restrictions.
OpenIFS builds 'out-of-source'; object (.o) and executables are not mixed with source code.

OpenIFS T21 test forecasts

In this section of the tutorial, we'll run the pre-compiled OpenIFS model on a simple T21 forecast.

You will:

learn about OpenIFS input and output files.
learn some switches to control OpenIFS.
learn how to run the model in parallel.
learn how to run the acceptability test.

Panel

title	Tasks - Examine t21test directory

Make sure you are in the directory: openifs//t21test.
List the files in this directory.

The directory t21test contains a number of files:

Set OpenIFS environment

Carry out the tasks above to connect to ccb and start an interactive session
Change into the model's main directory: cd perm/oifs43r3
Type the command: source ./oifs-config.ccb.sh

The oifs-config.ccb.sh script sets a number of Unix shell environment variables which define the type of OpenIFS compiled installation and location of files. These settings are specific to version 43r3.

If you do not run this setup script (or alternatively include its content in your Unix shell configuration file) then OpenIFS will not run correctly.

For more information on how to configure OpenIFS, please see the OpenIFS website: http://software.ecmwf.int/oifs/.

Panel

bgColor	#f0f0ff
title	Tasks - Examine the oifs43r3 directory

Make sure you are in the directory: ~/perm/oifs43r3/
List the files in this directory using the ls command

The directory contains a number of text files and directories:

Code Block

% ls
bin/     COPYING    LICENSE  oifs-config.ccb.sh  READMEs/
CHANGES  examples/  make/    python/             src/
CITE     fcm/       NOTICE

Code Block

% ls t21test
ICMGGepc8      ICMGGepc8INIUA  ICMSHepc8INIT  ifsdata  namelists
ICMGGepc8INIT  ICMSHepc8       README         job      ref_021_0144

Files beginning with 'ICM'.
These are the input files for this T21 experiment. They are in GRIB format. Do not move them from this directory. OpenIFS expects to find it's input files in the same directory as the main executable.

epc8 - this is the Experiment ID. Experiments IDs are used at ECMWF and initial conditions provided by ECMWF will always have an expt id.
ICMGGepc8 - 'GG' indicates these contain gridpoint fields.
ICMSHepc8 - 'SH' indicates these contains spherical harmonic fields.

job
Simple shell script to run the model. Described in more detail below.

ifsdata
Climate data fields used for T21 test integration. You should not move or rename this directory as the model will expect to find the climate files it needs in a directory of this name.

namelists
This file contains all of the input model fortran NAMELISTS. Not all of the namelists have their variables listed, only the variables commonly changed are listed here. Users should copy this file and modify it for the tests described below.

ref_021_0144
This file is reference output for the model tests. The model can be run in 'reference' mode where it checks it is working correctly by comparing some mathematical norms against these files. Reference runs are described in more detail under 'Acceptance testing' below.

Panel

title	Tasks - Examine grib files

Use the grib_ls and grib_dump commands to examine the contents of the ICM files.

Panel

title	Tasks - Run model

Run the model:

% ./job

What happens?

The model fails. Look in the NODE_001.01 file and find the subroutine traceback. Near the top of the traceback you will find the error messages.

Whenever the model fails, it will produce this traceback (controlled by DR_HOOK=1 in the job file).

Single process test

...

title	Tasks - Run the model as a single process

Copy the file namelists and run the model with a single task and single thread by executing the job script:

Code Block
% cp namelists fort.4 % ./job

t21test/

Directories:

src - contains all the source code for the model and supporting programs.
make - contains the build configuration files for the FCM compile command. Object files and executables will be in this directory organised according to the type of build (OIFS_BUILD environment variable).
t21test - self contained T21 model run for verifying the installation is correct.
ifsdata - contains additional input files for the model e.g. climatologies. Available as separate tarfiles.

Note:

OpenIFS has been precompiled on the HPCF.
All source code has been removed due to licensing restrictions.
OpenIFS builds 'out-of-source'; this means object (.o) files and executables (binary files) are not mixed with the source code.
The README file contains information about software requirements, setting up the local compilation environment, and where to get help and support.

OpenIFS T21 test forecasts

In this section of the tutorial, we will run the pre-compiled OpenIFS model on a simple T21 forecast.

You will:

learn about OpenIFS input and output files.
learn some switches to control OpenIFS.
learn how to run the model in parallel.
learn how to run the acceptability test.

Panel

bgColor	#f0f0ff
title	Tasks - Examine t21test directory

Make sure you are in the directory: ~/perm/oifs43r3/t21test
List the files in this directory.

The directory t21test contains a number of files:

Code Block
% ls t21test ICMGGepc8INIT ICMSHepc8INIT job ref_021_0072 ICMGGepc8INIUA ifsdata/ namelists run.ppn

Files beginning with 'ICM'.
These are the input files for this T21 experiment. They are in GRIB format. Do not move them from this directory. OpenIFS expects to find its input files in the same directory as the main executable.

epc8 - this is the Experiment ID. Experiments IDs are used at ECMWF and initial conditions provided by ECMWF will always have an expt id.
ICMGGepc8 - 'GG' indicates these contain gridpoint fields.
ICMSHepc8 - 'SH' indicates these contains spherical harmonic fields.

job
Shell script to run the model. Described in more detail below.

run.ppn
Simple shell script which calls job in an interactive shell environment.

ifsdata
Climate data fields used for T21 test integration. You should not move or rename this directory as the model will expect to find the climate files it needs in a directory of this name.

namelists
This file contains all of the input model fortran NAMELISTS. Not all of the namelists have their variables listed, only the variables commonly changed are listed here. Users should copy this file and modify it for the tests described below.

ref_021_0144
This file is reference output for the model tests. The model can be run in 'reference' mode where it checks it is working correctly by comparing some mathematical norms against these files. Reference runs are described in more detail under 'Acceptance testing' below.

Panel

bgColor	#f0f0ff
title	Tasks - Examine grib files

Use the grib_ls and grib_dump commands to examine the contents of the ICM files.

Panel

bgColor	#f0f0ff
title	Tasks - Run model

Run the model:

% ./run.ppn

What happens?

The model fails. Look at the standard output (or in the NODE_001.01 file when it is created) and find the subroutine traceback. Near the top of the traceback you will find the error messages.

Whenever the model fails, it will produce this traceback (controlled by DR_HOOK=1 in the job file).

Single process test

Panel

bgColor	#f0f0ff
title	Tasks - Run the model as a single process

Copy the file namelists to fort.4 and run the model with a single task and single thread by executing the job script:

Code Block
% cp namelists fort.4 % ./run.ppn

The model will expect to find a file called fort.4 in the same directory as the executable. This script copies the executable from make/cce-opt/oifs/bin.

If the run works you will see output like:

Code Block

...
  17:09:08 STEP    1 H=   0:10 +CPU=  0.276
           STEP    1 :## EC_MEMINFO    1 ccbppn01     309     206       0     16530     190    16080    1188     35807   61171      0.2   0.0 s/p
  17:09:08 STEP    2 H=   0:20 +CPU=  0.280
           STEP    2 :## EC_MEMINFO    1 ccbppn01     309     206       0     16530     190    16011    1188     35740   61170      0.2   0.0 s/p
  17:09:08 STEP    3 H=   0:30 +CPU=  0.268
           STEP    3 :## EC_MEMINFO    1 ccbppn01     309     206       0     16530     190    16008    1188     35734   61170      0.2   0.0 s/p
  17:09:09 STEP    4 H=   0:40 +CPU=  0.264
           STEP    4 :## EC_MEMINFO    1 ccbppn01     309     206       0     16530     190    15966    1188     35695   61170      0.2   0.0 s/p
  17:09:09 STEP    5 H=   0:50 +CPU=  0.268
           STEP    5 :## EC_MEMINFO    1 ccbppn01     309     206       0     16530     190    16008    1188     35734   61171      0.2   0.0 s/p
  17:09:09 STEP    6 H=   1:00 +CPU=  0.004
           STEP    6 :## EC_MEMINFO    1 ccbppn01     309     206       0     16530     190    16008    1188     35734   61171      0.2   0.0 s/p

This test runs only a small number of

The model will expect to find a file called fort.4 in the same directory as the executable. This script copies the executable from make/opt/bin.

If the run works you will see output like:

Code Block

...
signal_drhook(SIGSYS=31): New handler installed at 0x4d06cf; old preserved at 0x0
MPL_BUFFER_METHOD:  2           0
   16:03:46 STEP    0 H=   0:00 +CPU=  3.598
   16:03:46 STEP    1 H=   0:10 +CPU=  0.535
   16:03:47 STEP    2 H=   0:20 +CPU=  0.537
   16:03:48 STEP    3 H=   0:30 +CPU=  0.537
   16:03:48 STEP    4 H=   0:40 +CPU=  0.527
   16:03:49 STEP    5 H=   0:50 +CPU=  0.526
   16:03:49 STEP    6 H=   1:00 +CPU=  0.530

This test runs only 6 timesteps.

Model output

The model writes its output to a several files.

NODE_001.01 contains the text output (WRITE/PRINT statements). The numbers refer to task number and thread number. Only output from the master task & thread is normally output but this can be changed for debugging purposes.

ICM*epc8+0000 is the model output in GRIB format split into 2 files; one for the gridpoint, the other for spectral fields. These contain only a few output variables in this test. This file is a mix of GRIB1 and GRIB2 messages. See the Documentation for how to process this output.

ifs.stat is a small file that prints the model steps, time taken for each step and a 'norm' measure. This file can be usually ignored but is useful for debugging.

Panel

bgColor	#f0f0ff
title	Tasks - Examine model output

Look at the output from the model in the NODE_001.01 file from this successful run. Note the output of the model namelists and the statistics printed at the end. IFS has very comprehensive logging output which is useful for debugging and understanding the model's performance.

Look at the grib output files using the grib commands.

Parallel runs

These next short tests verify the model works correctly with either OpenMP parallel threading, MPI tasks and both and follow on from the serial tests above.

Panel

bgColor	#f0f0ff
title	Tasks - Enable OpenMP

Edit the file 'job' run.ppn and change the line: export OMP_NUM_THREADSNTHREADS=1 to export OMP_NUM_THREADSNTHREADS=2

Run ./job run.ppn as above.

Do the reported CPU times change?

Use the grib_ls command to look at grib output files - what do you notice?

...

If this works, look in the NODE_001.01 output file for the line:

...

to verify the model ran with 2 4 OpenMP threads.

Panel

bgColor	#f0f0ff
title	Tasks - Enable MPI

Edit the file 'job' run.ppn and change OMP_NUM_THREADS NTHREADS back to 1.

Change the line: NPROC=1 to NPROC=2.

Also, edit the fort.4 file and change NPROC to 2.

Rerun the job:
./jobrun.ppn

Do the reported CPU times change?

Note that increasing the number of tasks requires changing the number of tasks in two places.

Look in the NODE_001.01 output file for the line: "NUMBER OF TASKS 2NPROC = 2" to verify that two MPI tasks was used.

Mixed mode: OpenMP/MPI

The model can also be set to use NPROC=2 and OMP_NUM_THREADS and NTHREADS=2 to to use a total of 4 processes. However, this would require a computer with at least 4 cores to see any benefit and is not beneficial on the virtual machine.

Acceptance testing

The final step is to check the model is producing the numerical answers within acceptable limits, even if it runs the short tests above without failing. OpenIFS includes code that will compute internal statistical norms and compare against numbers supplied by ECMWF. The file: ref_021_0144 in the t21test directory contains statistical norms computed by the model run at ECMWF.

Panel

bgColor	#f0f0ff
title	Task - run acceptance test

To do the acceptance test, edit the namelists in fort.4 and look for the NAMCT0 namelist:

Code Block
&NAMCT0 LREFOUT=false, NSTOP=6,NAMCT0 LREFOUT=false

set the variable change the number of timesteps to 144 to run the model for 1 day (assuming you have not changed the default timestep of 10mins at T21) and set the LREFOUT to TRUE:

Code Block
&NAMCT0 LREFOUT=true, NSTOP=144,

With LREFOUT=true, at the last timestep OpenIFS will read the ref_021_0144 file and produce a new file: res_021_0144 (note the similar filenames!). The contents of the file should be similar to:

...

As long as the model reports 'calculations are correct' and the error is less than a few percent then the model is behaving satisfactorily in your compilation and run environment.

How to control model output

In this section, the main NAMELIST variables that control the output of the model are presented.

If you have time, try changing the variables, run the model and using the grib_ls and grib_dump commands to view the output grib file contents.

How to control output frequency

The namelist variables that determine the output from the model as it runs are:

Namelist : `NAMCT0`

LFPOS NFPOS - this should be set TRUE =2 in order to turn on model output and diagnostics.

...

NPOSTS & NHISTS - these are integer arrays that control the write times of the history files. They can be used for non-regular output intervals.

Examples

Regular output at fixed timesteps

NFRHIS=4, NFRPOS=4, NPOSTS=0, NHISTS=0,

...

Note
For this to work correctly, `NFRHIS * timestep` must equal an integer number of hours. The GRIB output will not work correctly if this isn't the case.

Non-regular output

NFRHIS=1,
NFRPOS=1,
NHISTS(0)=3,
NHISTS(1:3)=0,-3,-9,
NPOSTS(0)=-3,
NPOSTS(1:3)=0,-3,-9,

...

In this example, the model will write 3 separate output files at the first timestep (0hrs), 3hrs and 9hrs and then no more regardless of how long the model runs for.

How to change the output variables and post-processing

The namelist NAMFPC is the main namelist for the post-processing. Variables in this list can be sensitive to changes as many combinations are possible but not all work.

Model level output

To control model level output the following namelist variables (in NAMFPC) are used:

...

               e.g
               NFP3DFS = 5,
               MFP3DFS = 130, 135, 138, 155,
               would output the temperature (130), vert. vel. vertical velocities (135), relative vorticity (138), divergence (155) on model levels.

Panel

bgColor	#f0f0ff
title	Tasks - Change model output

Using the information above, try:

Adjusting the output frequency of the model.
Changing the list of model levels used for output and the output variables.

At the end of the session

Note
Do not forget to close your interactive ppn session on the HPCF using the exit command.

Space shortcuts

Page tree

Versions Compared

Old Version 20

New Version Current

Key

Introduction

In this tutorial...

What you will do:

At the end of this tutorial, you will:

Connecting to the ECMWF HPC Facilities

OpenIFS directories

OpenIFS directories

OpenIFS T21 test forecasts

Single process test

OpenIFS T21 test forecasts

Single process test

Model output

Parallel runs

Mixed mode: OpenMP/MPI

Acceptance testing

How to control model output

How to control output frequency

Namelist : `NAMCT0`

Examples

Regular output at fixed timesteps

Non-regular output

How to change the output variables and post-processing

Model level output

At the end of the session

Space shortcuts

Page tree

Page History

Versions Compared

Old Version 20

New Version Current

Key

Introduction

In this tutorial...

What you will do:

At the end of this tutorial, you will:

Connecting to the ECMWF HPC Facilities

OpenIFS directories

OpenIFS directories

OpenIFS T21 test forecasts

Single process test

OpenIFS T21 test forecasts

Single process test

Model output

Parallel runs

Mixed mode: OpenMP/MPI

Acceptance testing

How to control model output

How to control output frequency

Namelist : NAMCT0

Examples

Regular output at fixed timesteps

Non-regular output

How to change the output variables and post-processing

Model level output

At the end of the session

Namelist : `NAMCT0`