We recommend using IFS rather than OpenIFS on the ECMWF systems where possible. There is no scientific advantage to using OpenIFS compared to IFS.

The information provided here is for users with specific reasons to work with OpenIFS at ECMWF e.g. testing or transitioning work from OpenIFS to current versions of IFS.

Running OpenIFS

OpenIFS can be run as a standalone job on the ECMWF Cray in a similar way to how it would be run on an external system. Please see the ECMWF HPC support pages for more general information on how to run parallel jobs on the ECMWF Cray. It's also important to correctly set the environment. For more information see the following sections.

The example jobs illustrate how to run OpenIFS on a HPC batch system but note the PBS setup at ECMWF uses custom settings and jobs imported from external systems will need changes to the PBS settings. See support pages for more information.

OpenIFS can also be used in place of IFS in a standard prepIFS forecast 'fc' experiment. Doing so requires some script changes and specific settings in prepIFS (e.g. turn off the wave model which is not included with OpenIFS). For more details, see following sections.

For further help, please contact openifs-support@ecmwf.int.

Set environment

First correctly set the environment for OpenIFS before compiling the model. OpenIFS can run with either the Cray, GNU or Intel compilers.

Before loading the OpenIFS module, make sure you have the corresponding Cray PrgEnv module loaded (PrgEnv-cray is usually loaded by default).

Set the environment variable OIFS_HOME (e.g. export OIFS_HOME=$HOME/openifs) to the location of your OpenIFS installation before the module command.

Load modules
% module load PrgEnv-cray               #  may already be loaded
% module use --append ~openifs/modules  #  this could be placed in your .bashrc file
% module load openifs

This  module load command will provide the environment for the default version of OpenIFS, which may not be what you want.

To see the list of versions available for OpenIFS, run the module command:

% module avail openifs       #  make sure the module use --append.. command is run first (see above)

--------------------------- /home/rd/openifs/modules ------------------------
openifs/40r1     openifs/43r3      openifs/43r3.fdb (default)

To load different module rather than the default:

module load openifs/40r1

The openifs module will:

  • correctly load required packages: perl, python, ecCodes, fcm if required.
  • possibly set environment suitable for linking against the FDB library (see below) and load the FDB module (40r1 & 43r3.fdb only).

FDB: field database

The 40r1 and 43r3.fdb module will load the FDB module. This allows OpenIFS to write into MARS via the Field Database (FDB) library.  Once OpenIFS is compiled and linked with FDB, turn on FDB output by setting the LFDB=.true. and NOUTPUT=2 in the namelist file fort.4.

The openifs/43r3.fdb module loads the same modules as used for IFS 43r3 in prepIFS (it makes use of the ifs-support/43R3.7 module).

The openifs/43r3 module does not use the ifs-support module, nor load the FDB module. Instead it will load the modules required for OpenIFS depending on what compiler environment is set. This module is more useful for testing OpenIFS in different compilation environments.

Compiling OpenIFS on HPCF

Use an interactive session

Rather than compile the model on the Cray front-end nodes, it's better to use an interactive session to ask for resources and fast temporary disk. This improves compilation speed.

How to start interactive session on the ECMWF HPCF
% qsub -I -q df -l EC_total_tasks=12 -l EC_job_tmpdir=10G -l EC_memory_per_task=3G
qsub: waiting for job 6291795.ccbpar to start
qsub: job 6291795.ccbpar ready

and then proceed as described below as you normally would to compile the model.

The interactive session will normally last for 48hrs.

The 'df' and 'nf' queues are 'fractional' queues which only request part of a node. It is better to use these queues for low resolution testing or compilation. 'df' is the 'development' version of the 'nf' queue which usually gives a faster turnaround.

Note that the 'aprun' command can't be used in fractional queues. You need to add:

module load cray-snplauncher

to the job script and use mpiexec in place of aprun.

For more information please see: https://confluence.ecmwf.int/download/attachments/53514621/pbs_hpcf.pdf?api=v2

Compiling with the Cray compiler

The Cray compiler is recommended to build OpenIFS:

To compile with the Cray compiler set:
export OIFS_COMP=cce
export OFIS_BUILD=opt

If you don't have a copy of the OpenIFS code, retrieve it from the ftp site in the usual way (see User Guide).

You can also download the model from the OpenIFS git repository (called 'BitBucket') at ECMWF if you have permission (this is not general access but intended for developers and contributors to OpenIFS).

How to retrieve OpenIFS from the git repository & compile on ECMWF Cray
git clone ssh://git@git.ecmwf.int:7999/oifs/oifs43r3.git     # you must have permission to access the OpenIFS repository for this to work
cd oifs43r3v1
git checkout release/v1         # Switches to the version 2 release of OpenIFS at cycle 43r3.

export OIFS_HOME=$(pwd)

module use --append ~openifs/modules
module load openifs

oenv          # print out OpenIFS environment variables (alias set by openifs module)
omake         # build model (alias set by openifs module)

To run OpenIFS in place of IFS in a prepIFS experiment

The instructions below describe how to replace the IFS executable with OpenIFS, though IFS is still compiled and used to prepare the initial data. Only the forecast part of the experiment will use OpenIFS.

To ensure OpenIFS results are archived (written to MARS), it's necessary to link the model against the parallel I/O Field Database 'FDB' library (see above). The module load openifs/43r3.fdb will load the FDB library and set the correct environment variables to disable the dummy FDB code in OpenIFS.

The required changes are simple and made if the FDB module is loaded before the openifs module is loaded (as in the example above). Two environment variables are set before compilation of OpenIFS.

To turn on 'FDB' output rather than the normal ICM*+* files, set: LFDBOP 'On' in the prepIFS panel 'Main IFS control switches'. This will set LFDB=.true. and NOUTPUT=2 in the namelist NAMCT0.

To make prepIFS experiment run model for forecast instead of IFS:

  1. Create experiment as usual, using cycle 43r3. Add branch : nagc_CY43R3_oifs_run_in_prepifs or merge with other branch first.
  2. Compile OpenIFS as usual manually - the compilation of OpenIFS is not available as a task in suites created by prepIFS.
  3. Create directory: $SCRATCH/oifs_prepifs/<expid>/bin and copy master.exe into it.
  4. Make sure that the directory $SCRATCH/oifs_prepifs and all the subdirectories have read and execute access for all users (mode 755) .e.g. find oifs_prepifs -type d -exec chmod a+rx {} \;
  5. Make sure that the master.exe file has read and execute permission for everyone. e.g. chmod a+rx master.exe.
    These steps are necessary because the job on the Cray runs under a different userid.

As SMS events are not enabled in OpenIFS, the 'step' counter in the XCDP display will not update as the model is running only at the end of the forecast. However, the task will still fail correctly if the model fails.