Running OpenIFS

OpenIFS can be run as a standalone job on the ECMWF Cray in a similar way to how it would be run on an external system. Please see the ECMWF HPC support pages for more general information on how to run parallel jobs on the ECMWF Cray. It's also important to correctly set the environment. For more information see the following sections.

The example jobs illustrate how to run OpenIFS on a HPC batch system but note the PBS setup at ECMWF uses custom settings and jobs imported from external systems will need changes to the PBS settings. See support pages for more information.

OpenIFS can also be used in place of IFS in a standard prepIFS forecast 'fc' experiment. Doing so requires some script changes and specific settings in prepIFS (e.g. turn off the wave model which is not included with OpenIFS). For more details, see following sections.

For further help, please contact openifs-support@ecmwf.int.

Set environment

First correctly set the environment for OpenIFS before compiling the model.

OpenIFS can run with either the Cray, GNU or Intel compilers. Before loading the OpenIFS module, make sure you have the corresponding Cray PrgEnv module loaded (PrgEnv-cray is usually loaded by default).

Set the environment variable OIFS_HOME (e.g. export OIFS_HOME=$HOME/openifs) to the location of your OpenIFS installation before the module command.

% module load PrgEnv-cray
% module load fdb                        # needed if you want to run OpenIFS with prepIFS and put model results into the archive.
% module use --append ~openifs/modules
% module load openifs

The openifs module will:

• correctly load required packages: perl, python, grib_api, fcm if required.
• set OpenIFS environment variables: OIFS_CYCLE, OIFS_GRIB_API_DIR, OIFS_COMP, OIFS_BUILD, OIFS_DATA_DIR
• optionally set environment suitable for linking against the FDB library (see below) if the FDB module is loaded.

Compiling OpenIFS on HPCF

Use an interactive session

Rather than compile the model on the Cray front-end nodes, it's better to use an interactive session to ask for resources and fast temporary disk. This improves compilation speed.

% qsub -I -q df -l EC_total_tasks=12 -l EC_job_tmpdir=10G -l EC_memory_per_task=2G
qsub: waiting for job 6291795.ccbpar to start
qsub: job 6291795.ccbpar ready

and then proceed as described below as you normally would to compile the model.

The interactive session will last for 48hrs.

Compiling with the Cray compiler

The Cray compiler is recommended to build OpenIFS: export OIFS_COMP=cce.

If you don't have a copy of the OpenIFS code, retrieve it from the ftp site in the usual way (see User Guide).

You can also download the model from the OpenIFS git repository (called 'Stash') at ECMWF if you have permission (this is not general access but intended for developers and contributors to OpenIFS).

git clone ssh://git@software.ecmwf.int:7999/oifs/oifs40r1.git
cd oifs40r1
git checkout release/v2               # Switches to the version 2 release of OpenIFS at cycle 40r1.

export OIFS_HOME=$(pwd)

module load fdb
module use --append ~openifs/modules
module load openifs

oenv          # print out OpenIFS environment variables (alias set by openifs module)
omake         # build model (alias set by openifs module)

To run OpenIFS in place of IFS in a prepIFS experiment

Currently the instructions below describe how to replace the IFS executable with OpenIFS though IFS is still compiled and used to prepare the initial data. Only the forecast part of the experiment will use OpenIFS (this will be improved).

To ensure OpenIFS results are archived, it's necessary to link the model against the parallel I/O Field Database 'FDB' library.

The required changes are simple and made if the FDB module is loaded before the openifs module is loaded (as in the example above). Two environment variables are set before compilation of OpenIFS.

Check the following environment variables are set:

#  On cca/b, 
#  make fcm ignore the dummy fdb code as we'll link to the real library
% export OIFS_SRC_EXCL=openifs/dummy/dummy_fdb.f90

#  link to FDB library
% export OIFS_EXTRA_LIB="-L/usr/local/apps/fdb/4.6.3/CRAY/82/lib -lifsio"

To turn on 'FDB' output rather than the normal ICM*+* files, set: LFDBOP 'On' in the prepIFS panel 'Main IFS control switches'. This will set LFDB=.true. and NOUTPUT=2 in the namelist NAMCT0.

To make prepIFS experiment run model for forecast instead of IFS:

1. Create experiment as usual, cycle 40r1 is recommended. Add branch : nagc_CY40R1_oifs_run_in_prepifs or merge with other branch first.
2. Compile OpenIFS as usual manually - the compilation of OpenIFS is not available as a task in suites created by prepIFS.
3. Create directory: $SCRATCH/oifs_prepifs/<expid>/bin and copy master.exe into it.
4. Make sure that the directory $SCRATCH/oifs_prepifs and all the subdirectories have read and execute access for all users (mode 755) .e.g. find oifs_prepifs -type d -exec chmod a+rx {} \;
5. Make sure that the master.exe file has read and execute permission for everyone. e.g. chmod a+rx master.exe.
   These steps are necessary because the job on the Cray runs under a different userid.