This page describes the required steps to use OpenIFS/AC "v2ac-rc" on the ECMWF HPCF.

OpenIFS/AC compared to the standard OpenIFS model: An Overview

OpenIFS/AC consists of the standard OpenIFS model and the additional composition software package. Both components together are referred to as OpenIFS/AC (with AC standing for "atmospheric composition"). 

The standard OpenIFS model is described in detail in the OpenIFS User Guide. The user guide introduces the model's components and supported model grids. It informs about additional required software packages and how to build the OpenIFS model. It further describes how to carry out an acceptance test after a successful model build in order to verify the model's functionality.

Additional model sources

In the standard OpenIFS model the source code for atmospheric composition (atmospheric chemistry and aerosol code) has been replaced by dummy routines. The additional model sources for OpenIFS/AC replace these dummy routines and substitute the full composition source code. It is therefore necessary to rebuild the model executable after adding the additional model source code. This process is described further below.

Additional data packages

In order to run OpenIFS with atmospheric composition a substantial number of additional input data files are required. These provide initial conditions for the chemical and aerosol fields as well as additional boundary conditions for the model, mostly in the form of emission fluxes for a number of chemical tracers. These emissions are released mostly at the model surface, however they can also be associated with emission heights (e.g. for engine emissions from global air traffic). Further information is required for the physical removal of tracers through either wet scavenging or dry deposition, the latter of which is related to the surface vegetation type as used by the land surface scheme. 

The additional data packages required for atmospheric composition modelling need to be prepared for the model at the desired grid resolution. While the standard OpenIFS model can be used with a wide range of model grids, the prepared chemical input data is only available for few selected grid resolutions and for specific time periods. If the model is to be used for other time periods or at different horizontal or vertical grid resolutions then the additional data packages need first to be generated for this purpose. 

Obtaining and compiling the code

Working with the ftp server

In order to convert the standard OpenIFS model to OpenIFS/AC the additional software package for composition needs to be installed into an existing OpenIFS installation.

If both standard (meteorological) model and the atmospheric composition model are to be used at the same time, it is recommended to create a new OIFS_HOME directory for OpenIFS/AC and to first unpack the standard OpenIFS model sources into this directory (for instance, if the standard OpenIFS model is installed into directory 'oifs43r3', the new destination for the model sources could be extracted into directory 'openifs-ac').

Once the standard OpenIFS model code is unpacked, unpack package <package-name> into $OIFS_HOME, overwriting any pre-existing files from the standard model installation.

Working with the git repository

OpenIFS/AC is maintained in Philippe Le Sager's ECMWF git repository: https://git.ecmwf.int/users/nm6

git clone ssh://git@git.ecmwf.int/~nm6/oifs43r3.git
mv oifs43r3 openifs-ac
cd openifs-ac
git checkout v2ac-rc

Environment settings

In order to build OpenIFS/AC one can use the standard environment settings for oifs43r3v2 with the application of two changes:

• change OIFS_HOME to point to the installation of openifs-ac, and not to the standard oifs43r3v2 distribution
• change OIFS_DATA_DIR to a bespoke path as the additional data packages will be added to the climate data files
• before compiling the model sources edit $OIFS_HOME/make/oifs.fcm to enable chemistry (which is by default disabled)

It is recommend for the first two changes to use a specific "soifsac" alias to set the environment variables. Alternatively the settings can be enabled when running the prep scripts.

Later, for the t21 acceptance tests, the environment variables and paths are set within the oifs-config.<my_name>.sh which needs to be edited in the platform subdirectory. This is described below.

Obtaining additional data packages

• Download additional tarballs required for chemistry experiments:

cd $OIFS_DATA_DIR
module load ecfs
ecp ec:/nm6/oifs/oifs-ac-43r3-chem.tgz .  
ecp ec:/nm6/oifs/oifs-ac-43r3-xtra.tgz .
ecp ec:/nm6/oifs/oifs-ac-43r3-initcond.tgz .  (initial experiment data for first day of month, 12 months of 2010; goes into $OIFS_DATA_DIR/ICMCL)
ecp ec:/nm6/oifs/ICMCL-2010.tgz .

• Unpack the following files:  oifs-ac-43r3-chem.tgz and oifs-ac-43r3-xtra.tgz into $OIFS_DATA_DIR. This will result in the following directory structure:

The following files can be unpacked to a different location (outside $OIFS_DATA_DIR):

• Unpack oifs-ac-43r3-initcond.tgz into a directory INITIAL-CONDITIONS. These files are initial data files for the model experiments.
• Unpack ICMCL-2010.tgz into a directory ICMCL. This directory contains monthly files with surface boundary conditions (e.g. SSTs, soil temps, albedos, LAI etc).

(Comment:  if we unpack these files into $OIFS_DATA_DIR there is no need for a 43r3 subdirectory.  Note the ifsdata and rdata directories are missing and need to be copied over from a standard OIFS_DATA_DIR installation)

Building the model

It is recommended to open an interactive session on a MOM node for compilation:

% qsub -I -q dp -l EC_total_tasks=12 -l EC_job_tmpdir=10G -l EC_memory_per_task=3G

Compile the model as usual using fcm:

cd $OIFS_HOME/make

fcm make -v -j 6 -f oifs.fcm

The code should compile without errors when using CCE compilers. Make sure the binaries will later not be accidentally overwritten by any future compiler runs; we save them in a custom directory:

mv cce-opt v2ac-rc_cce-opt  (this is entirely optional)

Run the acceptance tests

The model package offers various tests to check the basic functionality of the model and to verify that all paths to the data directories are correct and accessible.

These tests are run on a T21 full Gaussian grid and are designed to require a minimum of computational resources. Several different tests are being offered which activate different pathways in the model source code. It is highly recommended to carry out these brief tests prior to using the model.

Step-by-step

To check the model is working normally carry out the various acceptance tests in $OIFS_HOME/t21test and t21test_xios_t255ac.

This requires editing the wrapper scripts job.sh and setup-exp.sh by setting the path to the appropriate master.exe and by changing RUNCMD to to something appropriate for the platform.

t21test:  This directory contains the original acceptance test which is provided with OpenIFS.  It should work in the same way for OpenIFS/AC as described in the OpenIFS User Guide.

t21test_xios:  This directory contains an acceptance test which makes use of the XIOS server – we do not use this here.

t21test_xios_t255ac:  The three acceptance tests in this directory make use of the additional compositon (AC) code and it is possible to run this test (a) without AC as in t21test, (b) with additional AC included, and (c) with both additional AC and using the XIOS server.

Now we can carry out the tests:

If the oifs_run script (or the oiac_run script) is used without the --nomove option it produces an error at the end of the acceptance test, stating the ICMGGepc8+* files cannot be found. This refers to the copying of model output to output1 and the postprocessing and is therefore not relevant to the model functionality.

Running OpenIFS/AC in Forecast or Climate Mode

The model is set up to run at Tl255L91 and it can be run in one of two modes:

Forecast mode:  The model carries out a brief fc experiment with daily restart files. The intention is to restart the model every 1 or 2 days with new meteorological initial data obtained from ea or od, while the chemical fields are retained from the restart file in order to simulate the evolution of the atmospheric chemistry with time.

Climate mode:  The model is run continuously for a longer period fc experiment while the dynamics is nudged every 6 hours to ei analyses (which were interpolated from 60 to 91 levels).

The experiment ID is set as oifc and the model is set up such that it runs on $SCRATCH of the HPCF.

$SCRATCH/43r3/prepAC/oifc/ will contain the initial and boundary conditions for the experiment.

$SCRATCH/43r3/runs/oifc/ will contain the experiment directory.

Generating the initial data for the experiment

See also the description here: Emission and IC preparation

Prior to running the model the initial experiment data needs to be generated. At present the model is set up to run from 2010-01-01 00:00 UTC. A series of scripts is run that collates the relevant data from MARS and other storage locations and compiles it in the appropriate format.

$OIFS_HOME/AC-experiments/ contains scripts to set up and carry out either a forecast or climate run.

Forecast experiments

The script to control the entire experiment workflow is forecast-workflow.sh which does the following:

• sets environment from scripts/oifs-config.${platform}.sh
• loads bash functions from scripts/lib_general.sh
• loads general experiment configuration parameters from config-fc.h
• creates experiment rundir on $SCRATCH space or clears it if necessary
• checks if this is an initial run leg or a restart leg, works out the leg of the next run
• submit initial experiment data preparation script forecast-prep.job to batch scheduler
• submit run script for next leg forecast-run.job to batch scheduler

Required modifications

module load sms ??

The following alterations were required from vanilla version in the git repository to make the process independent from Vincent's environment.
Lines with prefix - need to be deleted, lines with prefix + need to be added.

./forecast-prep.job:

-#PBS -l EC_billing_account=nlchekli

./forecast-run.job:

-#PBS -l EC_billing_account=nlchekli

./forecast-prep.sh:

- mv ICMCL-INIT-COMPO-${leg_start_date_yyyymmddhh}-${leg_end_date_yyyymmddhh} \
+ mv ${AC_DATA_DIR}/ICMCL-INIT-COMPO-${leg_start_date_yyyymmddhh}-${leg_end_date_yyyymmddhh} \

platform/oifs-config.ecmwf-cca.sh:

-export OIFS_HOME="/perm/ms/nl/nk9/oifs43r3"
+export OIFS_HOME="/perm/rd/damk/oifs/openifs-ac"

A larger number of code changes was required in the following files:

diff --git a/AC-experiments/config-fc.h b/AC-experiments/config-fc.h
index fd2f167..0c1c5cd 100644
--- a/AC-experiments/config-fc.h
+++ b/AC-experiments/config-fc.h
@@ -20,7 +20,7 @@
 EXP=oifc
 
 # Simulation start and end date. If setting time (not mandatory), **put it before** the date
-run_start_date="00:00 2013-07-01"
+run_start_date="00:00 2010-01-01"
 run_end_date="${run_start_date} + 2 days"
 
 # Restart frequency (a/k/a forecast length). Use any (reasonable) number and time unit you want.
@@ -56,19 +56,20 @@ LCHEM_OFFLINE_IC=false
 # a previous experiment
 #==========================================================
 # Experiment (can be different from the one used for METEO ICs)
-CHEM_INIEXP="b257"
+CHEM_INIEXP="b27y"
 CHEM_INIVER="ver15"             # default to CHEM_VER
 CHEM_INITYPE="fc"
 CHEM_INICLASS="nl"
 CHEM_INIFC=24                   # Time into the forecast to pickup IC  
-CHEM_INIDATE=2013070100         # Forecast start time. Default (if unset or 0) to run_start_date
-CHEM_INILEVELS="60"             # default to 60
+CHEM_INIDATE=2009123100         # Forecast start time. Default (if unset or 0) to run_start_date
+CHEM_INILEVELS="91"             # default to 60
 
-AERO_INIEXP="gbst"
+AERO_INIEXP="b27y"
 AERO_INITYPE="fc"
-AERO_INICLASS="rd"
-AERO_INIFC=744
-AERO_INIDATE=2012120100
+AERO_INICLASS="nl"
+AERO_INIFC=24
+AERO_INIDATE=2009123100
+AERO_INILEVELS="91"
 
 #==========================================================
 # Variables to retrieve METEO initial conditions from
@@ -78,15 +79,15 @@ AERO_INIDATE=2012120100
 # contains the right output. Note that MARS can regrid. Q: how flexible is that?
 
 INICLASS=nl
-INIEXPVER="b257"  # Also used for renaming output by get_gems_surface (aerosol sfc emiss), which is called by getgrb only => mot needed, do the renaming in getgrb!
+INIEXPVER="b27y"  # Also used for renaming output by get_gems_surface (aerosol sfc emiss), which is called by getgrb only => mot needed, do the renaming in getgrb!
 INISTREAM="OPER"
-BASEDATE="20130101" 
+BASEDATE="20100101" 
 BASETIME="00" 
-INILEVELS=60
+INILEVELS=91
 
 # model resolution we want to run
 RESOL=255
-LEVELS=60
+LEVELS=91
 GTYPE=l_2
 
 # Get Ozone from the METEO experiment
@@ -179,7 +180,7 @@ NSSALT=1
 LINJ_AER=false
 LVOLC_ALTI=false
 LAERSOA_CHEM=true
-LFIRE=false
+LFIRE=true
 
 # --- NAEAER
 LAERRRTM=false
@@ -213,7 +214,7 @@ WAVE=$( ${LWAM} && echo yes ||  echo no )
 # Variables configuring climate
 #==========================================================
 INICLASS_SST="od"
-CLMSST=yes
+CLMSST=no
 LBCSST=false
 PERSST=true
 FORCE_SST=no

diff --git a/AC-experiments/scripts/lib_icmcl_compo.sh b/AC-experiments/scripts/lib_icmcl_compo.sh
index d273e69..1741877 100644
--- a/AC-experiments/scripts/lib_icmcl_compo.sh
+++ b/AC-experiments/scripts/lib_icmcl_compo.sh
@@ -862,9 +862,9 @@ EOF
             rm InjectionHeight4IFS.grb InjectionHeight4IFShr.grb
         fi
         
-        if $LMCC_COMPO ; then
+        #if $LMCC_COMPO ; then
             cat  bcgf_int  omgf_int sogf_int  >>  aer_fire_emis 
-        fi
+        #fi
 
         rm -f  bcgf bcgf_int omgf omgf_int sogf sogf_int
     fi

diff --git a/AC-experiments/scripts/lib_initcond.sh b/AC-experiments/scripts/lib_initcond.sh
index 8df85cc..22b9e6e 100644
--- a/AC-experiments/scripts/lib_initcond.sh
+++ b/AC-experiments/scripts/lib_initcond.sh
@@ -258,20 +258,17 @@ prep_chem_init (){
gribIDsold=($($SCRIPTDIR/get_tablecol $tablefile_old grib))
gribIDs=($($SCRIPTDIR/get_tablecol $tablefile grib))

- # use aerosol scheme for SO4
- if [[ $LAERCHEM == true ]]; then
- gribso4=$($SCRIPTDIR/get_tablecol -w name=SO4 $tablefile_old grib)
- fi
-
for gribID in ${gribIDsold[*]}
do
chem_ml=$chem_ml/$gribID
done
chemml=$(echo $chem_ml|sed "s;//; ;")

- if [[ $LAERCHEM == true ]]; then 
- chemml=$(echo $chemml|sed "s;$gribso4/;;")
- fi
+ # if [[ $LAERCHEM == true ]]; then 
+ # for now always exclude SO4 from retrieval..
+ gribso4=$($SCRIPTDIR/get_tablecol -w name=SO4 $tablefile_old grib)
+ chemml=$(echo $chemml|sed "s;$gribso4/;;")
+ # fi

$MARS_CMD << EOF
retrieve,type=$typexp,levtype=ml,levelist=1/TO/$iclevels,date=$date_old,
@@ -282,6 +279,18 @@ prep_chem_init (){
expver=$expver,class=$class
EOF

+ # Add missing entry for SO4..
+ if [[ $LAERCHEM == false ]] ; then
+ # Get a zero-fiel
+ gribso4=$($SCRIPTDIR/get_tablecol -w name=SO4 $tablefile_old grib)
+ gribparam=$(printf %03d ${gribso4%.*})
+ gribtable=$(printf %03d ${gribso4#*.})
+ grib_copy -w paramId=210203 $gribfile_chem_final dumm_ic
+ grib_set -s paramId=${gribtable}${gribparam},scaleValuesBy=0.0 dumm_ic dumm_icb
+ cat dumm_icb >> $gribfile_chem_final
+ rm -rf dumm_ic dumm_icb
+ fi
+
# Correct grib number if different table files (ie CHEM_INIVER != CHEM_VER)
# 
# ! NOTE ! Assume that the species are in the same order in both tables!!
@@ -359,6 +368,14 @@ EOF
step=$fclen,param=${aeroml},grid=$grid,gaussian=reduced,target=$gribfile_aero_final,repres=gg,
expver=$expver,class=$class
EOF+                # In case LAERCHEM== false, make sure to include a dummy SO2 field
+                if [[ $LAERCHEM == false ]] ; then
+           info "generate dummy 210012 field (SO2-aer) "
+                  grib_copy -w paramId=210001 $gribfile_aero_final dumm_ic
+                  grib_set  -s paramId=210012,scaleValuesBy=0.0 dumm_ic dumm_icb
+                  cat dumm_icb >> $gribfile_aero_final
+                  rm -rf dumm_ic dumm_icb        
+         fi
                 # Now some further manipulation, to include aerosol sum (48.210) - not needed??
                 # simply get a zero-field?
                 grib_copy -w paramId=210001 $gribfile_aero_final dumm_ic