...

Running OpenIFS

OpenIFS can be run as a standalone job on the ECMWF Cray in a similar way to how it would be run on an external system. Please see the ECMWF HPC support pages for more general information on how to run parallel jobs on the ECMWF Cray. It's also important to correctly set the environment. For more information see the following sections.

...

For further help, please contact openifs-support@ecmwf.int.

Set environment

First correctly set the environment for OpenIFS before compiling the model. OpenIFS can run with either the Cray, GNU or Intel compilers.

...

• correctly load required packages: perl, python, grib_api, fcm if required.
• set OpenIFS environment variables: OIFS_CYCLE, OIFS_GRIB_DIR, OIFS_COMP, OIFS_BUILD, OIFS_DATA_DIR
• possibly set environment suitable for linking against the FDB library (see below) and load the FDB module (40r1 & 43r3.fdb only).

FDB: field database

The 40r1 and 43r3.fdb module will load the FDB module. This allows OpenIFS to write into MARS via the Field Database (FDB) library.  Once OpenIFS is compiled and linked with FDB, turn on FDB output by setting the LFDB=.true. and NOUTPUT=2 in the namelist file fort.4.

...

The openifs/43r3 module does not use the ifs-support module, nor load the FDB module. Instead it will load the modules required for OpenIFS depending on what compiler environment is set. This module is more useful for testing OpenIFS in different compilation environments.

Compiling OpenIFS on HPCF

Use an interactive session

Rather than compile the model on the Cray front-end nodes, it's better to use an interactive session to ask for resources and fast temporary disk. This improves compilation speed.

...

The interactive session will normally last for 48hrs.

Compiling with the Cray compiler

The Cray compiler is recommended to build OpenIFS:

...

git clone ssh://git@git.ecmwf.int:7999/oifs/oifs43r3.git     # you must have permission to access the OpenIFS repository for this to work
cd oifs43r3v1
git checkout release/v1         # Switches to the version 2 release of OpenIFS at cycle 43r3.

export OIFS_HOME=$(pwd)

module use --append ~openifs/modules
module load openifs

oenv          # print out OpenIFS environment variables (alias set by openifs module)
omake         # build model (alias set by openifs module)

To run OpenIFS in place of IFS in a prepIFS experiment

The instructions below describe how to replace the IFS executable with OpenIFS, though IFS is still compiled and used to prepare the initial data. Only the forecast part of the experiment will use OpenIFS.

...

To turn on 'FDB' output rather than the normal ICM*+* files, set: LFDBOP 'On' in the prepIFS panel 'Main IFS control switches'. This will set LFDB=.true. and NOUTPUT=2 in the namelist NAMCT0.

To make prepIFS experiment run model for forecast instead of IFS:

1. Create experiment as usual, using cycle 43r3. Add branch : nagc_CY43R3_oifs_run_in_prepifs or merge with other branch first.
2. Compile OpenIFS as usual manually - the compilation of OpenIFS is not available as a task in suites created by prepIFS.
3. Create directory: $SCRATCH/oifs_prepifs/<expid>/bin and copy master.exe into it.
4. Make sure that the directory $SCRATCH/oifs_prepifs and all the subdirectories have read and execute access for all users (mode 755) .e.g. find oifs_prepifs -type d -exec chmod a+rx {} \;
5. Make sure that the master.exe file has read and execute permission for everyone. e.g. chmod a+rx master.exe.
   These steps are necessary because the job on the Cray runs under a different userid.
   
   

...