Contents:

This page contains a quickstart guide for building and using OpenIFS 48r1 at model version 48R1.0.

We describe the process how to build the model from sources and carry out a few basic tests, how to run an example forecast experiment, for which initial experiment data is made available, and how to visualise the model output by creating plots.

In this version of the quickstart guide, it is assumed that the user has access to the ECMWF Atos HPC (hpc2020). This is however not a requirement and the description is kept as generic as possible and applies to different hardware platforms in the same way.

Extract the OpenIFS package

Create your local installation of OpenIFS 48R1.0 by cloning or branching from the git repository with the OpenIFS package. 

You will need approximately 4 GB of disk space for the model sources, the bundle packages, and the built model binaries. 

git clone ssh://git@git.ecmwf.int/~daha/openifs-main-git.git

Environment variables

For proper functionality the OpenIFS model requires a small number of Linux environment variables to be set.

The most important environment variable is OIFS_HOME, which describes the location of the OpenIFS model installation. This variable is required by all scripts used by the model.

Hence, the path where the git repository was extracted will need to be set to $OIFS_HOME. For example, if you clone the repository into your $HOME directory you should set:

export OIFS_HOME=$HOME/openifs-main-git/openifs_48R1.0/

This variable could be set in your shell configuration (e.g. in .bashrc) or by using a separate OpenIFS platform configuration script which could be loaded prior to using OpenIFS. We describe the set up of the platform configuration script in a later section on this page. 

 Build OpenIFS

In the next step the model binary executable (and other helper programs) will be built.

Requirements

The following software is required on your local system to build and use the OpenIFS model:

• Either Intel or GNU compilers are supported
• MPI, either MPICH or openmpi
• netCDF
• any others?

The software listed above is not part of the OpenIFS distribution and must be installed separately and prior to the building of OpenIFS. 

OpenIFS build system

In contrast to earlier model versions, the building of OpenIFS 48r1 is no longer based on the FCM configuration manager, but uses from now on the ecbuild ECMWF build system that is also used for the ECMWF IFS model, and which uses CMake at its core.

OpenIFS 48r1 is further distributed with a software bundle which automatically installs many required software packages during the build process, such as for instance ecbuild, ecCodes, metkit, etc. Hence, a separate installation of these libraries is no longer required as they have now become part of the OpenIFS distribution. 

Starting the build process

The  $OIFS_HOME/bin/openifs-test.sh  script can be used to build the model and run initial tests. 

• Edit the script and in line 16 set the location of your $OIFS_HOME directory which is the location where you have extracted the git repository and where this script should be located.
• The usage of the script is shown with the command:  openifs-test.sh -h
• The option -e defines the compiler environment (intel or gnu). The default is intel. 

Run the build process and the tests using the following command:

cd $OIFS_HOME
./bin/openifs-test.sh -cbt

The script builds the model binaries for the selected compiler and will run the ifs-test t21 and t42 tests.

  Set up the platform configuration file

You will need to set up a platform configuration file for your OpenIFS installation.

• This file can be located anywhere but the recommended location would be somewhere inside your $OIFS_HOME installation.
• This file sets global variables for your model installation, depending on your local system, and it will need to be "sourced" prior to running the model.
• We provide a template for this configuration file in $OIFS_HOME/bin/oifs48r1-config.sh. You should edit this file and update variable OIFS_HOME with your installation's path.

We recommend that you include the following command in your shell configuration file (e.g. .bashrc): 

source /path/to/configfile/oifs48r1-config.sh

This command should also be include in any batch job scripts that are intended to run OpenIFS.

 Set up a forecast experiment

An exmple forecast experiment has been prepared for OpenIFS 48r1. The experiment ID is i4xc. 

Extract the example forecast experiment i4xc.tgz into a location suitable for model experiments. 

Example: 
On the ECMWF hpc2020 our model installation $OIFS_HOME will be in $HOME/openifs and for the experiment we extract the i4xc package in $PERM. The experiment directory shall therefore be $PERM/i4xc/2016092500. 

Ensure the namelist files for the atmospheric model (fort.4) and for the wave model (wam_namelist) are found in the experiment directory.

You will need to copy two further scripts from the OpenIFS package into your experiment directory:

• oifs-run: this is a generic run script which executes the binary model program file.
• config.h: this is configuration file that determines settings for your experiment. It will be read by oifs-run. 

Copy both files from $OIFS_HOME/bin into your experiment directory.

cp /perm/openifs/oifs_data/48r1/example/i4xc.tgz $PERM
cd $PERM
tar -xvzf i4xc.tgz
cd ./i4xc/2016092500
cp $OIFS_HOME/bin/oifs-run .
cp $OIFS_HOME/bin/config.h .

Determine experiment parameters

You should edit the atmospheric model namelist file fort.4. It contains Fortran namelists which control model settings and switches. An important switch to edit is in namelist NAMRIP the variable CSTOP. Set this to the desired length of the forecast experiment. Experiment i4xc can be run for up to 144 hours (6 days) by setting CSTOP='h144'.

You should edit the config.h file which determines settings for this experiment. The oifs-run script will read the settings from this file. Alternatively, the settings can be passed to the oifs-run script via command line parameters, which takes precedence over the config.h settings. 

The config.h file contains the following settings:

OIFS_EXPID="i4xc"       # your experiment ID
OIFS_NPROC=8            # the number of MPI tasks
OIFS_NTHREAD=4          # the number of OpenMP threads
OIFS_GRIDTYPE="l"       # the grid type, either 'l' for linear reduced grid, or 'o' for the cubic octahedral grid
OIFS_RES="255"          # the spectral grid resolution
OIFS_NAMELIST='fort.4'  # the name of the atmospheric namelist file (the default is fort.4, so this line could be omitted)
OIFS_EXE="${OIFS_HOME}/build/bin/ifsMASTER.DP"  # the name and location of the model binary executable
OIFS_PPROC=true         # enable postprocessing of model output after the model run
OUTPUT_ROOT=$(pwd)      # folder where pproc output is created (only used if OIFS_PPROC=true). In this case an output folder is created in the experiment directory. 
LFORCE=false            # overwrite existing symbolic links in the experiment directory
LAUNCH=""               # the platform specific run command for the MPI environment (e.g. "mpirun", "srun", etc). 

Running the experiment

After all edits to namelists and config.h have been completed the model run can be started.

Depending on the available hardware experiments can either be run interactively or as a batch job.

In order to run the experiment interactively run oifs-run in your terminal. If no command line parameters are provided with the oifs-run command, then the values from the config.h will be used.

On the ECMWF hpc2020, running the model script interactively should be fine for lower grid resolutions up to T255L91.  If the LAUNCH variable in config.h remains empty (and no --runcmd parameter is provided in the command line) then the oifs-run script will use its default launch parameters:  srun -c${OIFS_NPROC} --mem=64GB --time=60  which will work fine for OIFS_NPROC=4 or 8 with experiment i4xc. 

Alternatively, a job script could be used for submitting the script to the batch scheduler. An example script run.atos.job for the SLURM batch scheduler, used on the ECMWF hpc2020 is provided here: $OIFS_HOME/bin/run.atos.job

This script should be edited as required. The LAUNCH command here is only "srun" without any further options, as the parallel environment settings are provided through the script headers.

# run interactively:
cd $PERM/i4xc
./oifs-run

# run as batch job:
cd $PERM/i4xc
sbatch run.atos.job

Postprocessing

If the OIFS_PPROC variable has been set to true (or if the --pproc command line parameter was used) then the model output in the experiment directory is further processed after completing the model run.

In this case the script will generate a folder called  output_YYYMMDD_HHMMSS , with YYYYMMDD being the current date and HHMMSS the current time.  This avoids accidental modification or overwriting of any previous results when the model experiment is repeated. The variable OUTPUT_ROOT in config.h determines where this ouput folder will be created. The default location is inside the experiment directory, but when assigning another path to OUTPUT_ROOT this could be created elsewhere.

The postprocessing groups all model output fields and diagnostics into individual GRIB files with ascending forecast time step. Also, a copy of the atmospheric model namelist file fort.4, as well as the ifs.stat and NODE.01_001 log files are moved into the output folder.

This postprocessing is required if the Metview Python script is to be used later to visualise the model output.

Plotting of model output

Here we describe in a brief summary how a small number of plots from the model results can be generated. This permits a first-order sanity check whether the model results look sensible.

For this we use the Metview graphics package developed at ECMWF. 

Copy the Metview processing code to your $PERM location:

cp /perm/openifs/oifs_data/48r1/example/mv.tgz $PERM
cd $PERM
tar -xvzf mv.tgz
cd mv

Now we process the OpenIFS model output into a dateset format that can be easily interpreted by Metview using a simplified plotting procedure.

Edit the file oifs_to_mv.sh and change the path variable:

• in_dir:  this needs to point to the folder where the postprocessed OpenIFS model experiment (from the previous section) is found. Absolute file paths are required for this to succeed. 

Afterwards, execute the script by running the command:  ./oifs_to_mv.sh.  This data conversion may take a couple of minutes to complete.

As a result regular gridded and compressed grib files are generated which can be visualised with by running the enclosed Jupyter Notebook  single.ipynb

Carry out the following steps:

• On the ECMWF Virtual Desktop Interface (VDI) open a terminal, log into the hpc2020 with command:  ssh hpc-login
• In the terminal start the Jupyter server on an interactive node, using the command:  ecinteractive -k
• After the interactive node has started you will be given a weblink to connect to the Jupyterlab session ("To manually re-attach go to <weblink>").
• Open a web browser (e.g. Chrome) inside the VDI and paste the weblink into the address field; this will connect to the Jupyter server.
• Find $PERM/mv/ipynb/single.ipynb in the browser, open the Notebook and run all cells of the Notebook.