...

This section describes the required steps to carry out a forecast experiment with chemistry. The essential three steps can be summarised as:

1. Edit paths and variables in $OIFS_HOME/AC-experiments/config.h. This controls model switches and the chemistry scheme. 
2. Prepare initial conditions using $OIFS_HOME/AC-epxeriments/prep-ic-icmcl-compo.sh.
3. Customise a job wrapper script for the relevant hardware platform (a *.job file) in order to submit the oifs-run.sh script to the batch scheduler.

...

• ecmwf-hpc2020.job.tmpl    –    run oifs without XIOS
• oifs-config.ecmwf-hpc2020.sh  –  template for the machine config file

The /prep-ic-icmcl-compo.sh/ script prepares the Initial Conditions,
  the climatology, and the atmospheric composition fluxes before you run
  OIFS. The main purpose is to create or copy three sets of files (IC,
  ICMCL and ICMCL-COMPO) needed to run an experiment. Paths to find
  those files are set in the /config.h/. See the section DATASETS in
  that file for details. Note that it is possible to use different data
  than those provide to test OpenIFS/AC but keep in mind this requires
  access to MARS archives at ECMWF.
  ┌────
  │  Usage:
  │         prep-ic-icmcl-compo.sh  START_DATE  END_DATE
  │
  │  Setup or create in $AC_IC_DIR:
  │
  │    - Initial Conditions for START_DATE (if not continuing/restarting a run)
  │         ICs are looked for in $OIFS_INIDATA_DIR. If not found, they are created.
  │
  │    - ICMCL climatology file for START_DATE to END_DATE.
  │         If $ICMCLDIR dir is set (in config.h), the ICMCL file is linked
  │         from $ICMCLDIR/ICMCL-yyyymm0100
  │         (Note: can contain data for dates before or after those required)
  │
  │    - ICMCL-COMPO (Emissions and Depositions) for START_DATE to END_DATE.
  │         ICMCL-COMPO are looked for in $OIFS_INIDATA_DIR. If not found, it is created.
  │
  │         (If \$AC_IC_DIR/ICMCL-INIT-COMPO-... exists, it will not be
  │         created again. This allows for faster re-run when developing.)
  │
  │  START_DATE, END_DATE:
  │    - Must be in a format understood by 'date -d' (eg 2010-12-15)
  │
  └────

  Note that:

  • If prepared beforehand, ICMCL and ICMCL-COMPO can contain data for dates
    before or after those requested, but filename formats expected BY THIS
    SCRIPT are:
    = ICMCL       : ICMCL-yyyymm0100
    = ICMCL-COMPO : ICMCL-INIT-COMPO or ICMCL-INIT-COMPO-START_DATE-END_DATE (with dates in YYYYMMDDhh)

  • $OIFS_INIDATA_DIR, $ICMCLDIR and $AC_IC_DIR are set in /config.h/

 Generating initial data overview

Initial conditions for the standard OpenIFS model can be generated using the OpenIFS Data Hub or alternatively they can be requested from the ECMWF OpenIFS support by email openifs-support@ecmwf.int. This provides initial experiment data and surface boundary conditions for the physical/meteorological NWP forecast model.

In order to simulate atmospheric composition further initial and boundary conditions are needed. These can be created using the prep-ic-icmcl-compo.sh script. 

• You need to edit variables inside the file config.h to correspond with the chosen settings for your experiment before running the script to generate the initial data.

At first the following environment variables need to be set to the appropriate locations. In our example described in the Section above these locations can be found in the OIFS_DATA_DIR under /data/openifs-ac: 

OIFS_ICMCL_DIR
This points to a directory containing initial conditions for the first day of all months of the year 2010 at Tl255L91. The ICMCL files contain boundary conditions (e.g. SSTs) for the entire month. 

OIFS_INIDATA_DIR_TM5
This points to a directory containing initial conditions required for the 'tm5' chemistry (two versions for Tl255L91 are available: one with AER aerosol and one without) for 2010-01-01. It also contains boundary conditions (emission fluxes and deposition velocities) for 2010-01-01 to 2010-01-02.

OIFS_INIDATA_DIR_CBA
As above but for the 'bascoetm5' chemistry. 

• Further settings that need changing in config.h are described below in the following section.
• Once all the changes have been made in the configuration file, the prep script can be run. Running the prep script will take a few minutes and will result in new data which will be written to folder chem_ic. This folder serves as a staging area for the initial experiment data. 
• After the prep script has completed it's run, the platform-specific job wrapper script can be executed to submit the model experiment job to the batch scheduler.

Example configuration settings for single forecast experiments

Select one of the following four T255L91 forecast configurations for your experiment.  The tested and supported configurations are:

...

91

...

Editing the experiment configuration file

For the forecast experiment to succeed and in order to generate the initial data correctly, you will need to edit variables inside the file config.h to correspond with the chosen settings for your experiment.

In config.h you should edit the following variables:

• Path variables need to be set to the appropriate locations. In our example described in the Section above these locations can be found in the OIFS_DATA_DIR under /data/openifs-ac: 

OIFS_ICMCL_DIR
This points to a directory containing initial conditions for the first day of all months of the year 2010 at Tl255L91. The ICMCL files contain boundary conditions (e.g. SSTs) for the entire month. 

OIFS_INIDATA_DIR_TM5
This points to a directory containing initial conditions required for the 'tm5' chemistry (two versions for Tl255L91 are available: one with AER aerosol and one without) for 2010-01-01. It also contains boundary conditions (emission fluxes and deposition velocities) for 2010-01-01 to 2010-01-02.

OIFS_INIDATA_DIR_CBA
As above but for the 'bascoetm5' chemistry. 

• Further variables and settings that need changing in config.h are described below in the next section for four possible forecast experiment configurations.
• Once all the changes have been made in config.h, the experiment initial and boundary condition files can be prepared as described in the following section. 
• After the data has been prepared the platform-specific job wrapper script can be executed to submit the model experiment job to the batch scheduler.

Example configuration settings for single forecast experiments

Select one of the following four T255L91 forecast configurations for your experiment.  The tested and supported configurations are:

Using TM5 chemistry without aerosol

We will run a 2 day forecast starting from 2010-01-01 with TM5 chemistry without aerosol.

Use the following settings in config.h

PLATFORM='ecmwf-atos'
RUNDIR=${SCRATCH}/43r3/experiments\${EXPID}\/runs  # where your experiment will run
AC_IC_DIR=${SCRATCH}/43r3/experiments\${EXPID}/chem_ic  # where your chemical initial conditions will be created
run_end_date="2010-01-03"

rst_freq='3 day'  # run for 3 days without restarts (no multi-leg exp)
USE_RESTART=false   # Do not write restart files (no multi-leg exp)
LCHEM=true
CHEM_SCHEME="tm5"
CHEM_VER="ver15"
OIFS_INIDATA_DIR=$path_to_directory_T255L91-tm5-none
AC_IC_ONCE=${OIFS_INIDATA_DIR}/ICMUAoiac.ac_rst'
MET_IC_DIR='${OIFS_ICMCL_DIR}/aabg/2010010100'

LAERO=false
LAERCHEM=false
LCHEM_AEROI=false
LINJ_CHEM=false
LINJ_AER=false
LAEROSFC=false

LWAM=true" config.h
LRLXG=false" config.h
LAC4IC=true" config.h

Using TM5 chemistry with AER

...

aerosol

We will run a 2 day forecast starting from 2010-01-01 with TM5 chemistry without and the AER bulk aerosol scheme.

Use the following settings in config.h

PLATFORM='ecmwf-atos'
RUNDIR=${SCRATCH}/43r3/experiments\${EXPID}\/runs  # where your experiment will run
AC_IC_DIR=${SCRATCH}/43r3/experiments\${EXPID}/chem_ic  # where your chemical initial conditions will be created
run_end_date="2010-01-03"

rst_freq='3 day'  # run for 3 days without restarts (no multi-leg exp)
USE_RESTART=false   # Do not write restart files (no multi-leg exp)
LCHEM=true
CHEM_SCHEME="tm5"
CHEM_VER="ver15"
OIFS_INIDATA_DIR=$path_to_directory_T255L91-tm5-noneaer
AC_IC_ONCE=${OIFS_INIDATA_DIR}/ICMUAoiac.ac_rst'
MET_IC_DIR='${OIFS_ICMCL_DIR}/aabg/2010010100'

LAERO=falsetrue
LAERCHEM=falsetrue
LCHEM_AEROI=falsetrue
LINJ_CHEM=falsetrue
LINJ_AER=falsetrue
LAEROSFC=falsetrue

LWAM=true" config.h
LRLXG=false" config.h
LAC4IC=true" config.h

Then run the prep script for initial conditions

./prep-ic-icmcl-compo.sh 2010-01-01 2010-01-03

...

.h

Using BASCOE-TM5 chemistry without aerosol

We will run a 2 day forecast starting from 2010-01-01 with combined BASCOE and TM5 chemistry and the AER bulk without aerosol scheme.

Use the following settings in config.h

PLATFORM='ecmwf-atos'
RUNDIR=${SCRATCH}/43r3/experiments\${EXPID}\/runs  # where your experiment will run
AC_IC_DIR=${SCRATCH}/43r3/experiments\${EXPID}/chem_ic  # where your chemical initial conditions will be created
run_end_date="2010-01-03"

rst_freq='3 day'  # run for 3 days without restarts (no multi-leg exp)
USE_RESTART=false   # Do not write restart files (no multi-leg exp)
LCHEM=true
CHEM_SCHEME="tm5"
CHEM_VER="ver15"
OIFS_INIDATA_DIR=$path_to_directory_T255L91-tm5-aer
AC_IC_ONCE=${OIFS_INIDATA_DIR}/ICMUAoiac.ac_rst'
MET_IC_DIR='${OIFS_ICMCL_DIR}/aabg/2010010100'

LAERO=true
LAERCHEM=true
LCHEM_AEROI=true
LINJ_CHEM=true
LINJ_AER=true
LAEROSFC=true

LWAM=true" config.h
LRLXG=false" config.h
LAC4IC=true" config.h

Then run the prep script for initial conditions

./prep-ic-icmcl-compo.sh 2010-01-01 2010-01-03

Using BASCOE-TM5 chemistry without aerosol

We will run a 2 day forecast starting from 2010-01-01 with combined BASCOE and TM5 chemistry without aerosol.

Use the following settings in config.h

PLATFORM='ecmwf-atos'
RUNDIR=${SCRATCH}/43r3/experiments\${EXPID}\/runs  # where your experiment will run
AC_IC_DIR=${SCRATCH}/43r3/experiments\${EXPID}/chem_ic  # where your chemical initial conditions will be created
run_end_date="2010-01-03"

rst_freq='3 day'  # run for 3 days without restarts (no multi-leg exp)
USE_RESTART=false   # Do not write restart files (no multi-leg exp)
LCHEM=true
CHEM_SCHEME="bascoetm5"
CHEM_VER="ver2f"
OIFS_INIDATA_DIR=$path_to_directory_T255L91-bascoetm5-none
AC_IC_ONCE=${OIFS_INIDATA_DIR}/ICMUAoiac.ac_rst'
MET_IC_DIR='${OIFS_ICMCL_DIR}/aabg/2010010100'

LAERO=false
LAERCHEM=false
LCHEM_AEROI=false
LINJ_CHEM=false
LINJ_AER=false
LAEROSFC=false

LWAM=true" config.h
LRLXG=false" config.h
LAC4IC=true" config.h

Then run the prep script for initial conditions

./prep-ic-icmcl-compo.sh 2010-01-01 2010-01-03

Using BASCOE-TM5 chemistry with AER aerosol

We will run a 2 day forecast starting from 2010-01-01 with combined BASCOE and TM5 chemistry without aerosol.

Use the following settings in config.h

PLATFORM='ecmwf-atos'
RUNDIR=${SCRATCH}/43r3/experiments\${EXPID}\/runs  # where your experiment will run
AC_IC_DIR=${SCRATCH}/43r3/experiments\${EXPID}/chem_ic  # where your chemical initial conditions will be created
run_end_date="2010-01-03"

rst_freq='3 day'  # run for 3 days without restarts (no multi-leg exp)
USE_RESTART=false   # Do not write restart files (no multi-leg exp)
LCHEM=true
CHEM_SCHEME="bascoetm5"
CHEM_VER="ver2f"
OIFS_INIDATA_DIR=$path_to_directory_T255L91-bascoetm5-none
AC_IC_ONCE=${OIFS_INIDATA_DIR}/ICMUAoiac.ac_rst'
MET_IC_DIR='${OIFS_ICMCL_DIR}/aabg/2010010100'

LAERO=true
LAERCHEM=true
LCHEM_AEROI=true
LINJ_CHEM=true
LINJ_AER=true
LAEROSFC=true

LWAM=true" config.h
LRLXG=false" config.h
LAC4IC=true" config.h

exp)
LCHEM=true
CHEM_SCHEME="bascoetm5"
CHEM_VER="ver2f"
OIFS_INIDATA_DIR=$path_to_directory_T255L91-bascoetm5-none
AC_IC_ONCE=${OIFS_INIDATA_DIR}/ICMUAoiac.ac_rst'
MET_IC_DIR='${OIFS_ICMCL_DIR}/aabg/2010010100'

LAERO=false
LAERCHEM=false
LCHEM_AEROI=false
LINJ_CHEM=false
LINJ_AER=false
LAEROSFC=false

LWAM=true" config.h
LRLXG=false" config.h
LAC4IC=true" config.h

Using BASCOE-TM5 chemistry with AER aerosol

We will run a 2 day forecast starting from 2010-01-01 with combined BASCOE and TM5 chemistry without aerosol.

Use the following settings in config.h

PLATFORM='ecmwf-atos'
RUNDIR=${SCRATCH}/43r3/experiments\${EXPID}\/runs  # where your experiment will run
AC_IC_DIR=${SCRATCH}/43r3/experiments\${EXPID}/chem_ic  # where your chemical initial conditions will be created
run_end_date="2010-01-03"

rst_freq='3 day'  # run for 3 days without restarts (no multi-leg exp)
USE_RESTART=false   # Do not write restart files (no multi-leg exp)
LCHEM=true
CHEM_SCHEME="bascoetm5"
CHEM_VER="ver2f"
OIFS_INIDATA_DIR=$path_to_directory_T255L91-bascoetm5-none
AC_IC_ONCE=${OIFS_INIDATA_DIR}/ICMUAoiac.ac_rst'
MET_IC_DIR='${OIFS_ICMCL_DIR}/aabg/2010010100'

LAERO=true
LAERCHEM=true
LCHEM_AEROI=true
LINJ_CHEM=true
LINJ_AER=true
LAEROSFC=true

LWAM=true" config.h
LRLXG=false" config.h
LAC4IC=true" config.h

Generating initial data overview

Initial conditions for the standard OpenIFS model can be generated using the OpenIFS Data Hub or alternatively they can be requested from the ECMWF OpenIFS support by email openifs-support@ecmwf.int. This provides initial experiment data and surface boundary conditions for the physical/meteorological NWP forecast model.

In order to simulate atmospheric composition further initial and boundary conditions are needed. These can be created using the prep-ic-icmcl-compo.sh script. 

The prep-ic-icmcl-compo.sh script (henceforth: prep script) carries out 3 tasks:

• Create a destination directory $AC_IC_DIR which serves as a staging area of all initial and boundary conditions required for the forecast experiment. Symbolic links will point later from inside the experiment run directory to these files.
• Copy the physical/meteorological initial conditions files that must be specific to the experiment start date from $OIFS_INIDATA_DIR to the new $AC_IC_DIR. If these files cannot be found then they are created. 
• Copy the physical boundary conditions file ICMCL{exp}INIT (SSTs, albedos, etc) from $OIFS_INIDATA_DIR to $AC_IC_DIR and if necessary truncate to length of experiment duration. 
• Copy the chemistry initial and boundary conditions file (emission fluxes, deposition velocities) ICMCL-INIT-COMPO from $OIFS_INIDATA_DIR to $AC_IC_DIR. If this file cannot be found it will be created. 

The syntax to run the prep script is:Then run the prep script for initial conditions

./prep-ic-icmcl-compo.sh 2010-01-01 2010-01-03<start_date> <end_date>

with start_date and end_date having the format YYYY-MM-DD. 

Running the prep script will take a few minutes and will result in new data which will be written to folder $AC_IC_DIR, by default $SCRATCH/43r3/experiments/{exp}/chem_ic. This folder serves as a staging area for the initial experiment data and if the target files already exist in this destination they they will not be overwritten. 

...

Multiple forecasts with restarts

...