Running OpenIFS

OpenIFS can be run as a standalone job on the ECMWF Cray in a similar way to how it would be run on an external system. Please see the ECMWF HPC support pages for more general information on how to run parallel jobs on the ECMWF Cray. It's also important to correctly set the environment. For more information see the following sections.

The example jobs illustrate how to run OpenIFS on a HPC batch system but note the PBS setup at ECMWF uses custom settings and jobs imported from external systems will need changes to the PBS settings. See support pages for more information.

OpenIFS can also be used in place of IFS in a standard prepIFS forecast 'fc' experiment. Doing so requires some script changes and specific settings in prepIFS (e.g. turn off the wave model which is not included with OpenIFS). For more details, see following sections.

For further help, please contact openifs-support@ecmwf.int.

Set environment

First correctly set the environment for OpenIFS before compiling the model. OpenIFS can run with either the Cray, GNU or Intel compilers.

Before loading the OpenIFS module, make sure you have the corresponding Cray PrgEnv module loaded (PrgEnv-cray is usually loaded by default).

Set the environment variable OIFS_HOME (e.g. export OIFS_HOME=$HOME/openifs) to the location of your OpenIFS installation before the module command.

% module load PrgEnv-cray               #  may already be loaded
% module use --append ~openifs/modules  #  this could be placed in your .bashrc file
% module load openifs

This  module load command will provide the environment for the default version of OpenIFS, which may not be what you want.

To see the list of versions available for OpenIFS, run the module command:

% module avail openifs       #  make sure the module use --append.. command is run first (see above)

--------------------------- /home/rd/openifs/modules ------------------------
openifs/40r1     openifs/43r3      openifs/43r3.fdb (default)

To load different module rather than the default:

module load openifs/40r1

The openifs module will:

• correctly load required packages: perl, python, ecCodes, fcm if required.
• set OpenIFS environment variables: OIFS_CYCLE, OIFS_GRIB_DIR, OIFS_COMP, OIFS_BUILD, OIFS_DATA_DIR
• possibly set environment suitable for linking against the FDB library (see below) and load the FDB module (40r1 & 43r3.fdb only).

FDB: field database

The 40r1 and 43r3.fdb module will load the FDB module. This allows OpenIFS to write into MARS via the Field Database (FDB) library.  Once OpenIFS is compiled and linked with FDB, turn on FDB output by setting the LFDB=.true. and NOUTPUT=2 in the namelist file fort.4.

The openifs/43r3.fdb module loads the same modules as used for IFS 43r3 in prepIFS (it makes use of the ifs-support/43R3.7 module).

The openifs/43r3 module does not use the ifs-support module, nor load the FDB module. Instead it will load the modules required for OpenIFS depending on what compiler environment is set. This module is more useful for testing OpenIFS in different compilation environments.

Compiling OpenIFS on HPCF

Use an interactive session

Rather than compile the model on the Cray front-end nodes, it's better to use an interactive session to ask for resources and fast temporary disk. This improves compilation speed.

% qsub -I -q df -l EC_total_tasks=12 -l EC_job_tmpdir=10G -l EC_memory_per_task=3G
qsub: waiting for job 6291795.ccbpar to start
qsub: job 6291795.ccbpar ready

and then proceed as described below as you normally would to compile the model.

The interactive session will normally last for 48hrs.

Compiling with the Cray compiler

The Cray compiler is recommended to build OpenIFS:

export OIFS_COMP=cce
export OFIS_BUILD=opt

If you don't have a copy of the OpenIFS code, retrieve it from the ftp site in the usual way (see User Guide).

You can also download the model from the OpenIFS git repository (called 'BitBucket') at ECMWF if you have permission (this is not general access but intended for developers and contributors to OpenIFS).

git clone ssh://git@git.ecmwf.int:7999/oifs/oifs43r3.git     # you must have permission to access the OpenIFS repository for this to work
cd oifs43r3v1
git checkout release/v1         # Switches to the version 2 release of OpenIFS at cycle 43r3.

export OIFS_HOME=$(pwd)

module use --append ~openifs/modules
module load openifs

oenv          # print out OpenIFS environment variables (alias set by openifs module)
omake         # build model (alias set by openifs module)

To run OpenIFS in place of IFS in a prepIFS experiment

The instructions below describe how to replace the IFS executable with OpenIFS, though IFS is still compiled and used to prepare the initial data. Only the forecast part of the experiment will use OpenIFS.

To ensure OpenIFS results are archived (written to MARS), it's necessary to link the model against the parallel I/O Field Database 'FDB' library (see above). The module load openifs/43r3.fdb will load the FDB library and set the correct environment variables to disable the dummy FDB code in OpenIFS.

The required changes are simple and made if the FDB module is loaded before the openifs module is loaded (as in the example above). Two environment variables are set before compilation of OpenIFS.

To turn on 'FDB' output rather than the normal ICM*+* files, set: LFDBOP 'On' in the prepIFS panel 'Main IFS control switches'. This will set LFDB=.true. and NOUTPUT=2 in the namelist NAMCT0.

To make prepIFS experiment run model for forecast instead of IFS:

1. Create experiment as usual, using cycle 43r3. Add branch : nagc_CY43R3_oifs_run_in_prepifs or merge with other branch first.
2. Compile OpenIFS as usual manually - the compilation of OpenIFS is not available as a task in suites created by prepIFS.
3. Create directory: $SCRATCH/oifs_prepifs/<expid>/bin and copy master.exe into it.
4. Make sure that the directory $SCRATCH/oifs_prepifs and all the subdirectories have read and execute access for all users (mode 755) .e.g. find oifs_prepifs -type d -exec chmod a+rx {} \;
5. Make sure that the master.exe file has read and execute permission for everyone. e.g. chmod a+rx master.exe.
   These steps are necessary because the job on the Cray runs under a different userid.