Note
Note that this documentation applies to the OpenIFS/AC code extension for OpenIFS 43r3. This does not apply to users of OpenIFS 48r1 or later model releases.

Table of Contents

...

This page describes how OpenIFS/AC v1 (derived from OpenIFS 43r3v2) can be obtained, built and used for forecast experiments. The documentation shows examples of how to use the model on the ECMWF Atos Sequana XH2000 HPCF (hpc2020). Adjustments will need to be made for local installations.

Info
Enquiries about OpenIFS/AC or technical questions should in the first instance be sent to openifs-support@ecmwf.int

OpenIFS/AC compared to the standard OpenIFS model

...

The standard OpenIFS model is described in detail in the OpenIFS 43r3 User Guide. The user guide introduces the model's components and supported model grids. It informs about additional required software packages and how to build the OpenIFS model. It further describes how to carry out an acceptance test after a successful model build in order to verify the model's functionality.

Info
Users who are new to OpenIFS should first read the OpenIFS 43r3 User Guide's sections 1 to 5 before proceeding with OpenIFS/AC, as the steps described therein are required for building OpenIFS/AC. The instructions in the user guide will not be repeated here.

...

No Format
ftp ftp.ecmwf.int (login with username and password for OpenIFS users) cd src/openifs-ac get openifs-ac.43r3.v1.tgzgz bye

Installing the OpenIFS/AC sources

...

Create a new home for the OpenIFS/AC installation. Here we call this location /home/openifs-ac.
Change into your new directory and unpack therein the original OpenIFS model sources: tar xvzf oifs43r3v2.tgz
In a second step, in the same folder unpack gunzip the OpenIFS/AC sources : tar xvzf and patch the original OpenIFS:
- gunzip openifs-ac.43r3.v1
.tgz
- .gz
- patch -p1 <openifs-ac.43r3.v1

Info
Note that some files of the original model will be replacedmodified. It is therefore essential to install the original OpenIFS 43r3v2 sources first before installing the OpenIFS/AC sources.

...

Change into your $OIFS_HOME directory where OpenIFS is installed: cd $OIFS_HOME
In this directory extract the downloaded source package: tar xvzf
- gunzip openifs-ac.43r3.v1
.tgz
- .gz
- patch -p1 <openifs-ac.43r3.v1

Info
Note that this unpacking will not only add new files to your existing $OIFS_HOME but it will also overwrite some of the existing files with updated versions required for OpenIFS/AC.

...

To do this we run the standard t21test for experiment epc8 (T21L19) with the setup-exp.sh new script.
The name for the hardware platform needs to be set (l.89, e.g. platform=ecmwf-hpc2020)
The test experiment runs in the RUNDIR test experiment runs in the RUNDIR directory specified in setup-exp.sh (l.94, the default is $SCRATCH/43r3/epc8/).
We also need to use the command line option -x and give as argument the absolute full path to the master.exe binary executable.
In setup-exp.sh you should change the value for RUN_NUMBER otherwise any pre-existing model output from this test will be overwritten.

No Format
./setup-exp.sh -n 16 -t 2 -x /home/damk/oifs/model/openifs-ac/make/gnu-opt-chem-fftw/oifs/bin/master.exe

...

/gnu-opt-chem-fftw/oifs/bin/master.exe

Test result: As before an error calculation file res_021_0072 is created in the run directory and the model output is found in the subfolder output1 (this name depends on the RUN_NUMBER variable).

Info

Running on the ECMWF HPCF:
This script requires more resources than are provided using the ecinteractive node.
One recommendation is to adapt an existing template from the platform subdirectory and submit this using the sbatch command.
Alternatively, the srun command could be used:
export NPROC=120
export OMP_NUM_THREADS=4
export OIFS_RUNCMD="srun --qos=np $OMP_NUM_THREADS -n $NPROC master.exe"
./setup-exp.sh -n 120 -t 4 -x master.exe

With chemistry: Now repeat the test with enabled model chemistry. This is a short 1-day test run that uses the newly added source code of OpenIFS/AC. It has a very limited set of options and should not be used for forecast experiments. Note that you cannot increase the experiment duration beyond one day because the boundary conditions are only supplied from 2013-07-01 to 2013-07-03.

...

No Format

AC-experiments/
├── config.h                   # MAIN experiment config file
├── ctrl/
│   ├── context_ifs.xml        # xios resource
│   ├── ifs_xml/               # xios resource
│   ├── iodef.xml              # xios resource
│   ├── namelistfc 
│   ├── namelist.ifs-epc8.sh   # namelist for t21test without chemistry
│   ├── namelist.ifs-oiac.sh   # namelist for t21test with chemistry
│   ├── namelist.ifs.sh        # namelist template for forecast experiments
│   ├── Table/                 # dir with AC tracers
│   └── wam_namelist
├── ecmwf-atoshpc2020.job             # job wrapper script for oifs-run.sh
├── oifs-run.sh                # MAIN run script
├── platform/                  # dir with platform-specific templates
├── prep_emis/                 # dir with emission utilities
├── prep-ic-icmcl-compo.sh     # MAIN script to prepare initial/boundary conditions
└── scripts/                   # dir with low-level scripts
    ├── add_nrt_fire_chem
    ├── gaussgr
    ├── get_alb_lai_intLimits
    ├── get_tablecol
    ├── grib_def.h
    ├── lib_chem_setup.sh
    ├── lib_general.sh           # lib of utilities
    ├── lib_icmcl_clima.sh       # lib of functs to create ICMCL boundary cond. files
    ├── lib_icmcl_compo.sh       # lib of functs to create ICMCL-COMPO files
    ├── lib_initcond_offline.sh
    └── lib_initcond.sh          # lib of functs to create initial conditions files

...

prep-ic-icmcl-compo.sh - a script to generate initial conditions for chemical fields for the selected run time. Begin and end date need to be given as an argument.
config.h - settings for model installation, paths, and experiment configuration.
oifs-run.sh - this script controls the workflow for the experiment: reads experiment config, sets up run directories (in $SCRATCH), loads platform configuration, edits namelists, links to initial conditions and executes OIFS_RUNCMD
ecmwf-atoshpc2020.job wrapper script for oifs-run.sh, required settings for the batch scheduler

...

it is necessary to create a platform-specific machine config file: oifs-config.ecmwf-atoshpc2020.sh

The above ecmwf-atoshpcs2020.job wrapper script to submit the batch job is based on templates in the platform directory; a number of templates exist and for the ECMWF HPCF the following templates are relevant:

...

No Format

PLATFORM='ecmwf-atoshpc2020'   # this needs to match the name of the platform configuration file
EXP=abcd                 # you should select here an experiment ID (4-letter alphanumeric word, must not start with '0')

RUNDIR=${SCRATCH}/43r3/experiments/${EXP}/runs         # this is the experiment directory
AC_IC_DIR=${SCRATCH}/43r3/experiments/${EXP}/chem_ic   # this is the stageingstaging directory for initial and boundary conditions data files

# Simulation start and end date. If setting time (not mandatory), **put it before** the date
run_start_date="00:00 2010-01-01"
run_end_date="${run_start_date} + 2 days"
 
rst_freq='6 day'        # Run for 6 days without restarts. Our experiment will finish after 2 days, so this will be ignored. 
USE_RESTART=false       # Do not write restart files.

...

Code Block

PLATFORM='ecmwf-atoshpc2020'

RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs'        # where your experiment will run
AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic'  # where your chemical initial conditions will be created
run_end_date="2010-01-03"
rst_freq='6 day'
USE_RESTART=false

LCHEM=true
CHEM_SCHEME="tm5"
CHEM_VER="ver15"
OIFS_INIDATA_DIR='${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-tm5-none'

LAERO=false
LAERCHEM=false
LCHEM_AEROI=false
LINJ_CHEM=false
LINJ_AER=false
LAEROSFC=false

LWAM=true
LRLXG=false

...

Code Block

PLATFORM='ecmwf-atoshpc2020'

RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs'        # where your experiment will run
AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic'  # where your chemical initial conditions will be created
run_end_date="2010-01-03"
rst_freq='6 day'
USE_RESTART=false

LCHEM=true
CHEM_SCHEME="tm5"
CHEM_VER="ver15"
OIFS_INIDATA_DIR='${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-tm5-aer'

LAERO=true
LAERCHEM=true
LCHEM_AEROI=true
LINJ_CHEM=true
LINJ_AER=true
LAEROSFC=true

LWAM=true
LRLXG=false

...

Code Block

PLATFORM='ecmwf-atoshpc2020'

RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs'        # where your experiment will run
AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic'  # where your chemical initial conditions will be created
run_end_date="2010-01-03"
rst_freq='6 day'
USE_RESTART=false

LCHEM=true
CHEM_SCHEME="bascoetm5"
CHEM_VER="ver2f"
OIFS_INIDATA_DIR='${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-bascoetm5-none'

LAERO=false
LAERCHEM=false
LCHEM_AEROI=false
LINJ_CHEM=false
LINJ_AER=false
LAEROSFC=false

LWAM=true
LRLXG=false

...

Code Block

PLATFORM='ecmwf-atoshpc2020'

RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs'        # where your experiment will run
AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic'  # where your chemical initial conditions will be created
run_end_date="2010-01-03"
rst_freq='6 day'
USE_RESTART=false

LCHEM=true
CHEM_SCHEME="bascoetm5"
CHEM_VER="ver2f"
OIFS_INIDATA_DIR='${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-bascoetm5-none'

LAERO=true
LAERCHEM=true
LCHEM_AEROI=true
LINJ_CHEM=true
LINJ_AER=true
LAEROSFC=true

LWAM=true
LRLXG=false

...

Space shortcuts

Page tree

Versions Compared

Old Version 144

New Version Current

Key

OpenIFS/AC compared to the standard OpenIFS model

Installing the OpenIFS/AC sources

Space shortcuts

Page tree

Page History

Versions Compared

Old Version 144

New Version Current

Key

OpenIFS/AC compared to the standard OpenIFS model

Installing the OpenIFS/AC sources