...

...

This page describes how OpenIFS/AC v1 (derived from OpenIFS 43r3v2) can be obtained, built and used for forecast experiments. The documentation shows examples of how to use the model on the ECMWF Atos Sequana XH2000 HPCF (hpc2020). Adjustments will need to be made for local installations. 

OpenIFS/AC compared to the standard OpenIFS model

...

The standard OpenIFS model is described in detail in the OpenIFS 43r3 User Guide. The user guide introduces the model's components and supported model grids. It informs about additional required software packages and how to build the OpenIFS model. It further describes how to carry out an acceptance test after a successful model build in order to verify the model's functionality.

...

ftp ftp.ecmwf.int
(login with username and password for OpenIFS users)
cd src/openifs-ac
get openifs-ac.43r3.v1.tgzgz
bye

Installing the OpenIFS/AC sources

...

• Create a new home for the OpenIFS/AC installation. Here we call this location /home/openifs-ac. 
• Change into your new directory and unpack therein the original OpenIFS model sources:   tar xvzf oifs43r3v2.tgz
• In a second step, in the same folder unpack gunzip the OpenIFS/AC sources :  tar xvzf and patch the original OpenIFS:
  
  • gunzip openifs-ac.43r3.v1
  .tgz
  • .gz
  • patch -p1 <openifs-ac.43r3.v1

...

• Change into your $OIFS_HOME directory where OpenIFS is installed:  cd $OIFS_HOME
• In this directory extract the downloaded source package:   tar xvzf
  • gunzip openifs-ac.43r3.v1
  .tgz
  • .gz
  • patch -p1 <openifs-ac.43r3.v1

...

• To do this we run the standard t21test for experiment epc8 (T21L19) with the setup-exp.sh new script.
• The name for the hardware platform needs to be set (l.89, e.g. platform=ecmwf-hpc2020)
• The test experiment runs in the RUNDIR test experiment runs in the RUNDIR directory specified in setup-exp.sh (l.94, the default is $SCRATCH/43r3/epc8/).
• We also need to use the command line option -x and give as argument the absolute full path to the master.exe binary executable.
• In setup-exp.sh you should change the value for RUN_NUMBER otherwise any pre-existing model output from this test will be overwritten. 

...

Test result:  As before an error calculation file res_021_0072 is created in the run directory and the model output is found in the subfolder output1 (this name depends on the RUN_NUMBER variable). 

With chemistry:  Now repeat the test with enabled model chemistry. This is a short 1-day test run that uses the newly added source code of OpenIFS/AC. It has a very limited set of options and should not be used for forecast experiments. Note that you cannot increase the experiment duration beyond one day because the boundary conditions are only supplied from 2013-07-01 to 2013-07-03. 

This experiment is named oiac and uses a Tl255L60 grid and runs for 24 hours. The default run directory is $SCRATCH/43r3/oiac/. 

./setup-exp.sh -c -n 16 -t 2 -x /home/damk/oifs/model/openifs-ac/make/gnu-opt-chem-fftw/oifs/bin/master.exe

This test runs with a 30 min time step and due to the computational requirements for the added chemistry it will take much longer to complete. The variable CHEM_SCHEME offers the choice of two chemistry schemes, the AER aerosol scheme is always enabled. 

Test result:  This experiment calculates budgets of atmospheric mass and ozone which are stored in the NODE.001_01 file and also in files $RUNDIR/output{$RUN_NUMBER}/massdia_chem_oiac.txt and ozonbud_chem_dia.txt.

These diagnostics can be compared with pre-calculated results in files massdia_chem__oiac_2013070100.txt and ozonbud_chem__oiac_2013070100.txt which are located in $OIFS_HOME/t21test_xois_t255ac/ctrl/. For example this could be done with a small shell script such as illustrated in the box below. 

EXPID=oiac; RUNDIR=${SCRATCH}/${OIFS_CYCLE}/${EXPID} ;  RUN_NUMBER=3  #checkoifs
fref1=${OIFS_HOME}/t21test_xios_ac/ctrl/massdia_chem__oiac_2013070100.txt
fref2=${OIFS_HOME}/t21test_xios_ac/ctrl/ozonbud_chem__oiac_2013070100.txt
fnew1=${RUNDIR}/output${RUN_NUMBER}/massdia_chem_oiac.txt
fnew2=${RUNDIR}/output${RUN_NUMBER}/ozonbud_chem_oiac.txt
diff $fref1 $fnew1
diff $fref2 $fnew2

Differences between the reference budgets and those from the t21test should be only small (below 5%). 

Single forecast experiments

This section describes the required steps to carry out a single forecast experiment (no restarts) with chemistry. The essential three steps can be summarised as:

1. Edit paths and variables in $OIFS_HOME/AC-experiments/config.h. This controls model switches and the chemistry scheme. 
2. Prepare initial conditions using $OIFS_HOME/AC-epxeriments/prep-ic-icmcl-compo.sh.
3. Customise a job wrapper script for the relevant hardware platform (a *.job file) in order to submit the oifs-run.sh script to the batch scheduler.

Overview of the AC-experiments directory

With chemistry:  Now repeat the test with enabled model chemistry. This is a short 1-day test run that uses the newly added source code of OpenIFS/AC. It has a very limited set of options and should not be used for forecast experiments. Note that you cannot increase the experiment duration beyond one day because the boundary conditions are only supplied from 2013-07-01 to 2013-07-03. 

This experiment is named oiac and uses a Tl255L60 grid and runs for 24 hours. The default run directory is $SCRATCH/43r3/oiac/. 

./setup-exp.sh -c -n 16 -t 2 -x /home/damk/oifs/model/openifs-ac/make/gnu-opt-chem-fftw/oifs/bin/master.exe

This test runs with a 30 min time step and due to the computational requirements for the added chemistry it will take much longer to complete. The variable CHEM_SCHEME offers the choice of two chemistry schemes, the AER aerosol scheme is always enabled. 

Test result:  This experiment calculates budgets of atmospheric mass, along with overall production and loss tendencies, as well as more detailed reaction budget terms relevant for the ozone production and loss tendency. These are printed in the NODE.001_01 file and afterwards extracted to the files $RUNDIR/output{$RUN_NUMBER}/massdia_chem_oiac.txt and ozonbud_chem_dia.txt.

These diagnostics can be compared with pre-calculated results in files massdia_chem__oiac_2013070100.txt and ozonbud_chem__oiac_2013070100.txt which are located in $OIFS_HOME/t21test_xois_t255ac/ctrl/. For example this could be done with a small shell script such as illustrated in the box below. 

EXPID=oiac; RUNDIR=${SCRATCH}/${OIFS_CYCLE}/${EXPID} ;  RUN_NUMBER=3  #checkoifs
fref1=${OIFS_HOME}/t21test_xios_ac/ctrl/massdia_chem__oiac_2013070100.txt
fref2=${OIFS_HOME}/t21test_xios_ac/ctrl/ozonbud_chem__oiac_2013070100.txt
fnew1=${RUNDIR}/output${RUN_NUMBER}/massdia_chem_oiac.txt
fnew2=${RUNDIR}/output${RUN_NUMBER}/ozonbud_chem_oiac.txt
diff $fref1 $fnew1
diff $fref2 $fnew2

As a rule of thumb, differences between the reference budget numbers and those from the t21test should be only small (below 5%), although this depends on the exact variable that is assessed. 

...

Single forecast experiments

This section describes the required steps to carry out a single forecast experiment (no restarts) with chemistry. The essential three steps can be summarised as:

1. Edit paths and variables in $OIFS_HOME/AC-experiments/config.h. This controls model switches and the chemistry scheme. 
2. Prepare initial conditions using $OIFS_HOME/AC-epxeriments/prep-ic-icmcl-compo.sh.
3. Customise a job wrapper script for the relevant hardware platform (a *.job file) in order to submit the oifs-run.sh script to the batch scheduler.

Overview of the AC-experiments directory

AC-experiments/
├── config.h 

AC-experiments/
├── config.h                           # MAIN experiment config file
├── ctrl/
│   ├── context_ifs.xml        # xios resource
│   ├── ifs_xml/               # xios resource
│   ├── iodef.xml              # xios resource
│   ├── namelistfc 
│   ├── namelist.ifs-epc8.sh   # namelist for t21test without chemistry
│   ├── namelist.ifs-oiac.sh   # namelist for t21test with chemistry
│   ├── namelist.ifs.sh        # namelist template for forecast experiments
│   ├── Table/                 # dir with AC tracers
│   └── wam_namelist
├── ecmwf-atoshpc2020.job             # job wrapper script for oifs-run.sh
├── oifs-run.sh                # MAIN run script
├── platform/                  # dir with platform-specific templates
├── prep_emis/                 # dir with emission utilities
├── prep-ic-icmcl-compo.sh     # MAIN script to prepare initial/boundary conditions
└── scripts/                   # dir with low-level scripts
    ├── add_nrt_fire_chem
    ├── gaussgr
    ├── get_alb_lai_intLimits
    ├── get_tablecol
    ├── grib_def.h
    ├── lib_chem_setup.sh
    ├── lib_general.sh           # lib of utilities
    ├── lib_icmcl_clima.sh       # lib of functs to create ICMCL boundary cond. files
    ├── lib_icmcl_compo.sh       # lib of functs to create ICMCL-COMPO files
    ├── lib_initcond_offline.sh
    └── lib_initcond.sh          # lib of functs to create initial conditions files

...

• prep-ic-icmcl-compo.sh  - a script to generate initial conditions for chemical fields for the selected run time. Begin and end date need to be given as an argument.
• config.h  - settings for model installation, paths, and experiment configuration.
• oifs-run.sh  - this script controls the workflow for the experiment: reads experiment config, sets up run directories (in $SCRATCH), loads platform configuration, edits namelists, links to initial conditions and executes OIFS_RUNCMD
• ecmwf-atoshpc2020.job  wrapper script for oifs-run.sh, required settings for the batch scheduler

...

• it is necessary to create a platform-specific machine config file:  oifs-config.ecmwf-atoshpc2020.sh

The above ecmwf-atoshpcs2020.job wrapper script to submit the batch job is based on templates in the platform directory; a number of templates exist and for the ECMWF HPCF the following templates are relevant:

...

PLATFORM='ecmwf-atoshpc2020'   # this needs to match the name of the platform configuration file
EXP=abcd                 # you should select here an experiment ID (4-letter alphanumeric word, must not start with '0')

RUNDIR=${SCRATCH}/43r3/experiments/${EXP}/runs         # this is the experiment directory
AC_IC_DIR=${SCRATCH}/43r3/experiments/${EXP}/chem_ic   # this is the stageingstaging directory for initial and boundary conditions data files

# Simulation start and end date. If setting time (not mandatory), **put it before** the date
run_start_date="00:00 2010-01-01"
run_end_date="${run_start_date} + 2 days"
 
rst_freq='6 day'        # Run for 6 days without restarts. Our experiment will finish after 2 days, so this will be ignored. 
USE_RESTART=false       # Do not write restart files.

...

PLATFORM='ecmwf-atoshpc2020'

RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs'        # where your experiment will run
AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic'  # where your chemical initial conditions will be created
run_end_date="2010-01-03"
rst_freq='6 day'
USE_RESTART=false

LCHEM=true
CHEM_SCHEME="tm5"
CHEM_VER="ver15"
OIFS_INIDATA_DIR='${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-tm5-none'

LAERO=false
LAERCHEM=false
LCHEM_AEROI=false
LINJ_CHEM=false
LINJ_AER=false
LAEROSFC=false

LWAM=true
LRLXG=false

...

PLATFORM='ecmwf-atoshpc2020'

RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs'        # where your experiment will run
AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic'  # where your chemical initial conditions will be created
run_end_date="2010-01-03"
rst_freq='6 day'
USE_RESTART=false

LCHEM=true
CHEM_SCHEME="tm5"
CHEM_VER="ver15"
OIFS_INIDATA_DIR='${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-tm5-aer'

LAERO=true
LAERCHEM=true
LCHEM_AEROI=true
LINJ_CHEM=true
LINJ_AER=true
LAEROSFC=true

LWAM=true
LRLXG=false

...

PLATFORM='ecmwf-atoshpc2020'

RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs'        # where your experiment will run
AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic'  # where your chemical initial conditions will be created
run_end_date="2010-01-03"
rst_freq='6 day'
USE_RESTART=false

LCHEM=true
CHEM_SCHEME="bascoetm5"
CHEM_VER="ver2f"
OIFS_INIDATA_DIR='${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-bascoetm5-none'

LAERO=false
LAERCHEM=false
LCHEM_AEROI=false
LINJ_CHEM=false
LINJ_AER=false
LAEROSFC=false

LWAM=true
LRLXG=false

...

PLATFORM='ecmwf-atoshpc2020'

RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs'        # where your experiment will run
AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic'  # where your chemical initial conditions will be created
run_end_date="2010-01-03"
rst_freq='6 day'
USE_RESTART=false

LCHEM=true
CHEM_SCHEME="bascoetm5"
CHEM_VER="ver2f"
OIFS_INIDATA_DIR='${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-bascoetm5-none'

LAERO=true
LAERCHEM=true
LCHEM_AEROI=true
LINJ_CHEM=true
LINJ_AER=true
LAEROSFC=true

LWAM=true
LRLXG=false

...

Due to the much increased computational requirements for simulating atmospheric composition, forecast experiments are often broken into individual parts (sometimes referred to as "legs"). For each leg a restart of the forecast experiment needs to be carried out. 

Model Output
The oifs-run.sh script for OpenIFS/AC has been designed to be able to cope with restarts or "legs", and the model output from each leg a restart of the forecast experiment needs to be carried out. 

Model Output
The oifs-run.sh script for OpenIFS/AC has been designed to be able to cope with restarts or "legs", and the model output from each experiment leg is stored in individual subfolders named by leg number. In the case of a single forecast experiment without restarts the output will be written to $RUNDIR/output/001.

Log files
Log file and experiment-related configuration data (namelist, NODE file, ifs.stat), as well as text files with diagnostic model outuput (e.g. mass and ozone budgets) are copied to $RUNDIR/log/001.

Restart Data
Model data relevant for restarts (not discussed here on this page) will be stored in $RUNDIR/restart/${next_leg_number}.

XIOS output
Files relevant to XIOS server output are stored in $RUNDIR/ifs_xml (not discussed here on this page). 

Multiple forecasts with restarts

...

experiment leg is stored in individual subfolders named by leg number. In the case of a single forecast experiment without restarts the output will be written to $RUNDIR/output/001.

Log files
Log file and experiment-related configuration data (namelist, NODE file, ifs.stat), as well as text files with diagnostic model outuput (e.g. mass and ozone budgets) are copied to $RUNDIR/log/001.

Restart Data
Model data relevant for restarts (not discussed here on this page) will be stored in $RUNDIR/restart/${next_leg_number}.

XIOS output
Files relevant to XIOS server output are stored in $RUNDIR/ifs_xml (not discussed here on this page). 

...

Multiple forecasts with restarts

There are several reasons why a user may wish to restart a forecast experiment. The increase in computational costs for simulating atmospheric composition may require to break a longer forecast experiment into individual sections ("legs"). Also, a restart will allow to initialise the meteorology in the model with data taken from an analysis of the current modelled time. 

Due to the time scales associated with the different chemical processes and the lifetime of chemical tracers with respect to their loss processes, it is often convenient to retain the chemical tracer fields in the model simulation and update only the driving meteorology at the time of the restart. In some applications, the model could even be initialised every single day with an up-to-date meteorological analysis field while the chemical fields are left to evolve.  The previous section described how chemical initial conditions can be created at the end of an experiment run, thus retaining the chemical tracer information from the previous experiment leg. 

We will here provide some examples of the principle steps required for restarting experiments.

Example 1:  Longer experiment with manual stop and restart

The experiment parameters need to be defined by editing config.h. The following settings are used for a 2-month experiment with a stop-and-restart after the first month. This will result in two experiment legs of 1 month duration each.

run_start_date="2010-01-01"
run_end_date="${run_start_date} + 2 months"
rst_freq="1 month"
USE_RESTART=true

The initial conditions for each leg need to be either created or gathered from a prepared repository.

prep-ic-icmcl-compo.sh  2010-01-01 2010-02-01
prep-ic-icmcl-compo.sh  2010-02-01 2010-03-01 

These processes can be run simultaneously. The meteorological fields and initial conditions can be requested from the OpenIFS Data Hub. The wrapper script needs to be submitted for the first experiment leg. Once the experiment has completed, config.h needs to be edited to use the initial experiment data from the date and time when the first leg has stopped. Then the wrapper script for the second month should be submitted.

Example 2: Using a workflow script

The 4 jobs in the previous example, can be all triggered automatically and in a timely fashion with a single script. The  $OIFS_HOME/AC-experiments/platform/<platform-name>-workflow.sh templates provide examples that need to be adjusted for the local hardware and file paths. In this case the config.h file only needs editing once and instead of the batch job wrapper script the workflow script is submitted. 

This option is at an experimental development stage and it is therefore currently not supported for general users.

...