This page describes how OpenIFS/AC v1 (derived from OpenIFS 43r3v2) can be obtained, built and used for forecast experiments. The documentation shows examples of how to use the model on the ECMWF Atos Sequana XH2000 HPCF. Adjustments will need to be made for local installations.
Enquiries about OpenIFS/AC or technical questions should in the first instance be sent to openifs-support@ecmwf.int
OpenIFS/AC compared to the standard OpenIFS model
OpenIFS/AC consists of the standard OpenIFS model and the additional composition software package. Both components together are referred to as OpenIFS/AC (with AC standing for "atmospheric composition").
The standard OpenIFS model is described in detail in the OpenIFS User Guide. The user guide introduces the model's components and supported model grids. It informs about additional required software packages and how to build the OpenIFS model. It further describes how to carry out an acceptance test after a successful model build in order to verify the model's functionality.
Users who are new to OpenIFS should first read the OpenIFS User Guide's sections 1 to 5 before proceeding with OpenIFS/AC, as the steps described therein are required for building OpenIFS/AC. The instructions in the user guide will not be repeated here.
Additional model sources
In the standard OpenIFS model the source code for atmospheric composition (atmospheric chemistry and aerosol code) has been replaced by dummy routines. The additional model sources for OpenIFS/AC replace these dummy routines and substitute the full composition source code. It is therefore necessary to rebuild the binary model executable after adding the additional model source code. This process is described further below.
Additional data packages
In order to run OpenIFS with atmospheric composition a substantial number of additional input data files are required. These provide initial conditions for the chemical and aerosol fields as well as additional boundary conditions for the model, mostly in the form of emission fluxes for a number of chemical tracers. These emissions are released mostly at the model surface, however they can also be associated with emission heights (e.g. for engine emissions from global air traffic). Further information is required for the physical removal of tracers through either wet scavenging or dry deposition, the latter of which is related to the surface vegetation type.
The additional data packages required for atmospheric composition modelling need to be prepared for the model at the desired grid resolution. While the standard OpenIFS model can be used with a wide range of model grids, the prepared chemical input data is only available for few selected grid resolutions and for specific time periods. If the model is to be used for other time periods or at different horizontal or vertical grid resolutions then the additional data packages need first to be generated for this purpose.
Note that the additional data packages required for OpenIFS/AC are only made available for specific time periods and grid resolutions.
Obtaining the model code
This section contains information about where the source code package for OpenIFS/AC can be obtained and how the sources should be installed.
Working with the ftp server
You need to connect to the ECMWF ftp server to download the source package for OpenIFS/AC. If your institution has signed the OpenIFS license agreement then you can request the ftp password by emailing openifs-support@ecmwf.int.
You can either convert your existing OpenIFS installation or create a new installation for OpenIFS/AC. We recommend to create a new installation and for this you will also need the sources of the standard OpenIFS model (oifs43r3v2) which can also be found on the ftp server.
How to establish a connection to the ECMWF ftp server and how to obtain OpenIFS 43r3 sources is described in more detail here in the OpenIFS User Guide.
ftp ftp.ecmwf.int (login with username and password for OpenIFS users) cd src/openifs-ac get openifs-ac.43r3.v1.tgz bye
Installing the OpenIFS/AC sources
Installing the model sources and building the model (including acceptance tests) requires approximately 3-4 GB of disk space. The additionally required climatological data packages are not yet included; their installation is described below and will require further disk space.
Method 1: We recommend to create a new installation for OpenIFS/AC. This will preserve your original OpenIFS installation and will allow you to use the standard (meteorological) model and the atmospheric composition model interchangeably.
To proceed you should:
- Create a new home for the OpenIFS/AC installation. Here we call this location /home/openifs-ac.
- Change into your new directory and unpack therein the original OpenIFS model sources: tar xvzf oifs43r3v2.tgz
- In a second step, in the same folder unpack the OpenIFS/AC sources: tar xvzf openifs-ac.43r3.v1.tgz
Note that some files of the original model will be replaced. It is therefore essential to install the original OpenIFS 43r3v2 sources first before installing the OpenIFS/AC sources.
Method 2: Alternatively, in order to convert the standard OpenIFS model to OpenIFS/AC the additional software package for composition needs to be installed into an existing OpenIFS installation. You can create separate binary executables, with and without chemistry enabled. We recommend to create first a backup of your original installation before proceeding.
For the conversion you should:
- Change into your $OIFS_HOME directory where OpenIFS is installed: cd $OIFS_HOME
- In this directory extract the downloaded source package: tar xvzf openifs-ac.43r3.v1.tgz
Note that this unpacking will not only add new files to your existing $OIFS_HOME but it will also overwrite some of the existing files with updated versions required for OpenIFS/AC.
Obtaining additional data packages
In addition to the enhancements of the model sources, OpenIFS/AC also requires additional external data to carry out forecast experiments (e.g. climatological data for boundary conditions, fluxes for emitted chemical compounds, etc). This data will complement the data found usually in the $OIFS_DATA_DIR location. The data volume for the standard distribution of OpenIFS/AC is approximately 21 GB.
Note: It is recommended to create a new OIFS_DATA_DIR directory specifically for use with OpenIFS/AC which is kept separate from the data directory for the meteorological model.
As a next step you should:
- Create a new directory which will contain the additional data that is required for atmospheric composition modelling and it will also contain the new
OIFS_DATA_DIR
. Here we call this location /data/openifs-ac. Download the required four additional data tgz archives into /data/openifs-ac. The following commands will work on the Atos HPC:
cd /data/openifs-ac module load ecfs ecp ec:/nm6/oifs/oifs-ac-43r3-chem.tgz . ecp ec:/nm6/oifs/oifs-ac-43r3-xtra.tgz . ecp ec:/nm6/oifs/oifs-ac-43r3-initcond.tgz . ecp ec:/nm6/oifs/ICMCL-2010.tgz .
Unpack oifs-ac-43r3-chem.tgz and oifs-ac-43r3-xtra.tgz into /data/openifs-ac. This will result in the following directory structure inside /data/openifs-ac :
43r3/chemistry
AC specific 43r3/xdata
AC specific 43r3/rtables
Standard dataset for OpenIFS 43r3 43r3/climate.v015
Standard dataset for OpenIFS 43r3 + AC specifics 43r3/ifsdata
Standard dataset for OpenIFS 43r3 43r3/wam
Wave model data for OpenIFS 43r3, needs to be added explicitly
The following two tgz archives can be unpacked to any location, however in this example the files we will also extract them into /data/openifs-ac:
- Unpack oifs-ac-43r3-initcond.tgz into /data/openifs-ac which results in a directory /data/openifs-ac/INITIAL-CONDITIONS.
The files therein are initial data files for the first day of month for all 12 months of 2010. - Unpack ICMCL-2010.tgz into /data/openifs-ac which results in a directory /data/openifs-ac/ICMCL.
This directory contains monthly files with surface boundary conditions (e.g. SSTs, soil temps, albedos, LAI etc). - In the tgz archives above, the data for the year 2013, provided on a Tl255L60 grid, is required for the T21 acceptance tests. The data for year 2010 is provided on a Tl255L91 grid and can be used for the example forecasts.
- From your conventional OpenIFS installation's
OIFS_DATA_DIR
copy the directories ifsdata and rtables to /data/openifs-ac/43r3 as they are also needed here. If this is your first model installation, these data directories are also available from the ftp server. - Further, create a directory /data/openifs-ac/43r3/wam which needs to contain wave model initial data for 2010-01-01.
Building the model
This section describes how to build the binary model executable for OpenIFS/AC. Most problems related to building the model executable are due to incorrect environment settings. Therefore we recommend that particular care should be given to the environment settings to ensure that they match the local hardware platform.
Environment settings
Similar to the standard OpenIFS model, OpenIFS/AC requires system and user software libraries (e.g. eccodes, netcdf, etc) and relies on shell environment variables which need to be set according to your local installation. In order to avoid the duplication of documentation we refer to the OpenIFS User Guide as the prerequisites for OpenIFS are also valid for OpenIFS/AC. For more information about environment variable settings please see here.
We recommend to start from a successfully tested local environment for OpenIFS and work from there to make the necessary adjustments for OpenIFS/AC.
In order to build OpenIFS/AC the standard environment settings for oifs43r3v2 can be used with the application of two changes:
- Change $OIFS_HOME to point to your installation of openifs-ac (and not to the standard oifs43r3 installation), in our example this is /home/openifs-ac.
- Change $OIFS_DATA_DIR to the location with the additional data packages, in our example this is /data/openifs-ac.
- Before compiling the model sources edit $OIFS_HOME/make/oifs.fcm to enable chemistry (which is by default disabled).
It is recommend to apply the first two changes in $OIFS_HOME/oifs-config.editme.sh and to activate the settings with the command: source ./oifs-config.editme.sh
before attempting to build the model.
In order to run OpenIFS/AC a number of further changes and additions to the environment are required. These are described below.
Building the binary executable
On the ECMWF HPCF it is possible to compile the model using a login node. However, in order to complete the acceptance testing with chemistry an interactive node should be requested.
% ecinteractive -c 12 -m 32GB -t 06:00:00
If you have a configuration script that sets the appropriate environment for OpenIFS you should run this first before building the model executables.
In your command line you should type
export OIFS_CHEM=enable
to overwrite the default setting in oifs.fcm (or alternatively you can edit the variable in file oifs.fcm)
Further you should make a choice whether you want to build with XIOS server support, in that case set appropriate environment variables. Otherwise set
export OIFS_XIOS=disable
to disable XIOS.
Then compile the model using fcm (here with verbose output and with 16 parallel child processes):
cd $OIFS_HOME/make fcm make -v -j 16 -f oifs.fcm
You should only proceed further after the code has compiled and all the executables were built without errors.
Running the acceptance tests
The model package offers various tests to check the basic functionality of the model and to verify that all paths to the data directories are correct and accessible.
Overview
The acceptance tests are run on a T21 full Gaussian grid and are designed to require a minimum of computational resources. Several different tests are being offered which activate different pathways in the model source code. It is highly recommended to carry out these brief tests to verify that the model is working normally prior to using the model for forecast experiments.
The tests are located in $OIFS_HOME/t21test
and t21test_xios_t255ac
.
Running the tests for the first time requires editing the wrapper scripts job.sh
and setup-exp.sh
by setting the path to the appropriate master.exe
and by changing RUNCMD to to something appropriate for the platform.
t21test: This directory contains the original acceptance test which is provided with OpenIFS. It should work in the same way for OpenIFS/AC as described in the OpenIFS User Guide.
t21test_xios: This directory contains an OpenIFS acceptance test which makes use of the XIOS server – we do not describe this here.
t21test_xios_t255ac: This directory contains the new tests for OpenIFS/AC. The test consists of a short 1-day run to demonstrate the working of OpenIFS/AC. It has a very limited set of options. Note that you cannot increase the runtime of the experiment because the provided ICMCL file does not contain data for an extended model run. The three acceptance tests in this directory make use of the additional compositon (AC) code and it is possible to run this test (a) without AC as in t21test, (b) with additional AC included, and (c) for both of the previous tests additionally XIOS server output can be selected.
Note: If the XIOS server is to be used in these tests then a precompiled version of its libraries needs to be accessible and the appropriate paths need to be set in oifs-config.{platform}.sh. We do not describe here how XIOS is built, more information on XIOS can be found here.
Note: During the acceptance test with enabled chemistry the following error messages from the Rosenbrock solver can occur at times. In the t21test a small number of occurrences of this error can be ignored:
Forced exit from Rosenbrock due to the following error:
--> No of steps exceeds maximum bound
T= 1172.85033258064 and H= 9.36605529265499
Step-by-step
Here we describe the required steps to carry out the tests.
Using t21test
./job.sh -n 12 -t 2 -m ../make/gnu-opt-chem-fftw/oifs/bin/master.exe
If the oifs_run script (or the oiac_run script) is used without the --nomove option it produces an error at the end of the acceptance test, stating the ICMGGepc8+* files cannot be found. This refers to the copying of model output to output1 and the postprocessing and is therefore not relevant to the model functionality.
Test result: This test runs in directory $OIFS_HOME/t21test
and model output is written into subfolder output1
. The model run also produces the file res_021_0072
which contains results from the error calculation. This error should be below 5%. For more information see the OpenIFS User Guide.
Using t21test_xios_t255ac
Without chemistry: This test uses a different script compared to the standard OpenIFS model's t21test. First we will verify its correct functionality by running the test without activating the model chemistry.
- To do this we run the standard t21test for experiment epc8 (T21L19) with the
setup-exp.sh
new script. - The test experiment runs in the
RUNDIR
directory specified insetup-exp.sh
(the default is$SCRATCH/43r3/epc8/
). - We also need to use the command line option -x and give as argument the absolute full path to the
master.exe
binary executable. - In
setup-exp.sh
you should change the value forRUN_NUMBER
otherwise any pre-existing model output from this test will be overwritten.
./setup-exp.sh -n 16 -t 2 -x /home/damk/oifs/model/openifs-ac/make/gnu-opt-chem-fftw/oifs/bin/master.exe
Test result: As before an error calculation file res_021_0072
is created in the run directory and the model output is found in the subfolder output1
(this name depends on the RUN_NUMBER variable).
With chemistry: Now repeat the test with enabled model chemistry. This experiment is named oiac
and uses a T255L60 grid and runs for 24 hours. The default run directory is $SCRATCH/43r3/oiac/
.
./setup-exp.sh -c -n 16 -t 2 -x /home/damk/oifs/model/openifs-ac/make/gnu-opt-chem-fftw/oifs/bin/master.exe
This test runs with a 30 min time step and due to the computational requirements for the added chemistry it will take much longer to complete. The variable CHEM_SCHEME offers the choice of two chemistry schemes, the AER aerosol scheme is always enabled.
Note: If the XIOS server is to be used in the t21test experiments (option -i
) then the variables OIFS_XIOS
and OIFS_XIOS_DIR
need to be set appropriately in the configuration file in the platform
directory. Also, the binary executable master.exe
needs to have been built with XIOS support enabled.
Test result: This experiment calculates budgets of atmospheric mass and ozone which are stored in the NODE.001_01
file and also in files $RUNDIR/output{$RUN_NUMBER}/massdia_chem_oiac.txt
and ozonbud_chem_dia.txt
. These diagnostics can be compared with pre-calculated results in files massdia_chem__oiac_2013070100.txt
and ozonbud_chem__oiac_2013070100.txt
which are located in $OIFS_HOME/t21test_xois_t255ac/ctrl/
.
Single forecast experiments
This section describes the required steps to carry out a forecast experiment with chemistry. The sequence of steps can be summarised as
(1) generate initial conditions or use existing initial data,
(2) set up experiment parameters in config.h and select the appropriate batch job wrapper script for the HPC hardware,
(3) submit the experiment and monitor progress.
Preparing an experiment
AC-experiments directory overview
The experiment controls are in $OIFS_HOME/AC-experiments/
There the following scripts can be found:
- prep-ic-icmcl-compo.sh - a script to generate initial conditions for chemical fields for the selected run time. Begin and end date need to be given as an argument.
- config.h - settings for model installation, paths, and experiment configuration.
- oifs-run.sh - this script controls the workflow for the experiment: reads experiment config, sets up run directories (in $SCRATCH), loads platform configuration, edits namelists, links to initial conditions and executes OIFS_RUNCMD
- ecmwf-atos.job wrapper script for oifs-run.sh, required settings for the batch scheduler
In $OIFS_HOME/AC-experiments/platform:
- it is necessary to create a machine config file: oifs-config.ecmwf-atos.sh
The above ecmwf-atos.job wrapper script to submit the batch job is based on templates in the platform directory; the following templates exist:
- ecmwf-hpc2020.job.tmpl – run oifs without XIOS
- oifs-config.ecmwf-hpc2020.sh – template for the machine config file
Generating initial data overview
Initial conditions for the standard OpenIFS model can be generated using the OpenIFS Data Hub or alternatively they can be requested from the ECMWF OpenIFS support by email openifs-support@ecmwf.int. This provides initial experiment data and surface boundary conditions for the physical/meteorological NWP forecast model. In order to simulate atmospheric composition further initial and boundary conditions are needed. These can be created using the prep-ic-icmcl-compo.sh script.
Note: The prep-ic-icmcl-compo.sh script requires external software: the CDO command line operators to manipulate and analyse climate and NWP model data which are available here: https://code.mpimet.mpg.de/projects/cdo/.
- You need to edit variables inside the file
config.h
to correspond with the chosen settings for your experiment before running the script to generate the initial data.
At first the following environment variables need to be set to the appropriate locations. In our example described in the Section above these locations can be found in the OIFS_DATA_DIR
under /data/openifs-ac:
OIFS_ICMCL_DIR
This points to a directory containing initial conditions for the first day of all months of the year 2010 at Tl255L91. The ICMCL files contain boundary conditions (e.g. SSTs) for the entire month.
OIFS_INIDATA_DIR_TM5
This points to a directory containing initial conditions required for the 'tm5' chemistry (two versions for Tl255L91 are available: one with AER aerosol and one without) for 2010-01-01. It also contains boundary conditions (emission fluxes and deposition velocities) for 2010-01-01 to 2010-01-02.
OIFS_INIDATA_DIR_CBA
As above but for the 'bascoetm5' chemistry.
- Further settings that need changing in
config.h
are described below in the following section. - Once all the changes have been made in the configuration file, the prep script can be run. Running the prep script will take a few minutes and will result in new data which will be written to folder
chem_ic
. This folder serves as a staging area for the initial experiment data. - After the prep script has completed it's run, the platform-specific job wrapper script can be executed to submit the model experiment job to the batch scheduler.
Example configuration settings for single forecast experiments
Select one of the following four configurations for your experiment.
Using TM5 chemistry without aerosol
We will run a 2 day forecast starting from 2010-01-01 with TM5 chemistry without aerosol.
Use the following settings in config.h
PLATFORM='ecmwf-atos' RUNDIR=${SCRATCH}/43r3/experiments\${EXPID}\/runs # where your experiment will run AC_IC_DIR=${SCRATCH}/43r3/experiments\${EXPID}/chem_ic # where your chemical initial conditions will be created run_end_date="2010-01-03" rst_freq='3 day' # run for 3 days without restarts (no multi-leg exp) USE_RESTART=false # Do not write restart files (no multi-leg exp) LCHEM=true CHEM_SCHEME="tm5" CHEM_VER="ver15" OIFS_INIDATA_DIR=$path_to_directory_T255L91-tm5-none AC_IC_ONCE=${OIFS_INIDATA_DIR}/ICMUAoiac.ac_rst' MET_IC_DIR='${OIFS_ICMCL_DIR}/aabg/2010010100' LAERO=false LAERCHEM=false LCHEM_AEROI=false LINJ_CHEM=false LINJ_AER=false LAEROSFC=false LWAM=true" config.h LRLXG=false" config.h LAC4IC=true" config.h
Then run the prep script for initial conditions
./prep-ic-icmcl-compo.sh 2010-01-01 2010-01-03
Using TM5 chemistry with AER aerosol
We will run a 2 day forecast starting from 2010-01-01 with TM5 chemistry and the AER bulk aerosol scheme.
Use the following settings in config.h
PLATFORM='ecmwf-atos' RUNDIR=${SCRATCH}/43r3/experiments\${EXPID}\/runs # where your experiment will run AC_IC_DIR=${SCRATCH}/43r3/experiments\${EXPID}/chem_ic # where your chemical initial conditions will be created run_end_date="2010-01-03" rst_freq='3 day' # run for 3 days without restarts (no multi-leg exp) USE_RESTART=false # Do not write restart files (no multi-leg exp) LCHEM=true CHEM_SCHEME="tm5" CHEM_VER="ver15" OIFS_INIDATA_DIR=$path_to_directory_T255L91-tm5-aer AC_IC_ONCE=${OIFS_INIDATA_DIR}/ICMUAoiac.ac_rst' MET_IC_DIR='${OIFS_ICMCL_DIR}/aabg/2010010100' LAERO=true LAERCHEM=true LCHEM_AEROI=true LINJ_CHEM=true LINJ_AER=true LAEROSFC=true LWAM=true" config.h LRLXG=false" config.h LAC4IC=true" config.h
Then run the prep script for initial conditions
./prep-ic-icmcl-compo.sh 2010-01-01 2010-01-03
Using BASCOE-TM5 chemistry without aerosol
We will run a 2 day forecast starting from 2010-01-01 with combined BASCOE and TM5 chemistry without aerosol.
Use the following settings in config.h
PLATFORM='ecmwf-atos' RUNDIR=${SCRATCH}/43r3/experiments\${EXPID}\/runs # where your experiment will run AC_IC_DIR=${SCRATCH}/43r3/experiments\${EXPID}/chem_ic # where your chemical initial conditions will be created run_end_date="2010-01-03" rst_freq='3 day' # run for 3 days without restarts (no multi-leg exp) USE_RESTART=false # Do not write restart files (no multi-leg exp) LCHEM=true CHEM_SCHEME="bascoetm5" CHEM_VER="ver2f" OIFS_INIDATA_DIR=$path_to_directory_T255L91-bascoetm5-none AC_IC_ONCE=${OIFS_INIDATA_DIR}/ICMUAoiac.ac_rst' MET_IC_DIR='${OIFS_ICMCL_DIR}/aabg/2010010100' LAERO=false LAERCHEM=false LCHEM_AEROI=false LINJ_CHEM=false LINJ_AER=false LAEROSFC=false LWAM=true" config.h LRLXG=false" config.h LAC4IC=true" config.h
Then run the prep script for initial conditions
./prep-ic-icmcl-compo.sh 2010-01-01 2010-01-03
Using BASCOE-TM5 chemistry with AER aerosol
We will run a 2 day forecast starting from 2010-01-01 with combined BASCOE and TM5 chemistry without aerosol.
Use the following settings in config.h
PLATFORM='ecmwf-atos' RUNDIR=${SCRATCH}/43r3/experiments\${EXPID}\/runs # where your experiment will run AC_IC_DIR=${SCRATCH}/43r3/experiments\${EXPID}/chem_ic # where your chemical initial conditions will be created run_end_date="2010-01-03" rst_freq='3 day' # run for 3 days without restarts (no multi-leg exp) USE_RESTART=false # Do not write restart files (no multi-leg exp) LCHEM=true CHEM_SCHEME="bascoetm5" CHEM_VER="ver2f" OIFS_INIDATA_DIR=$path_to_directory_T255L91-bascoetm5-none AC_IC_ONCE=${OIFS_INIDATA_DIR}/ICMUAoiac.ac_rst' MET_IC_DIR='${OIFS_ICMCL_DIR}/aabg/2010010100' LAERO=true LAERCHEM=true LCHEM_AEROI=true LINJ_CHEM=true LINJ_AER=true LAEROSFC=true LWAM=true" config.h LRLXG=false" config.h LAC4IC=true" config.h
Then run the prep script for initial conditions
./prep-ic-icmcl-compo.sh 2010-01-01 2010-01-03
Multiple forecasts with restarts
This option is at an experimental development stage and it is therefore currently not supported for general users.
Its implementation depends on scripting that is dependant on the hardware platform. Suggestions for script templates that are used on the ECMWF HPCF can be provided on request.
Further information
----- THE REMAINING TEXT IS TO BE DELETED PRIOR TO PUBLICATION -----
EADME
━━━━━━━━━━━━━━━━━━━━━━━━━
ATMOSPHERIC COMPOSITION
━━━━━━━━━━━━━━━━━━━━━━━━━
Table of Contents
─────────────────
1. Test data
.. 1. Datasets for OIFS/AC
.. 2. Overview table of provided Initial Conditions
2. t21test_xios_t255ac
.. 1. Presentation
.. 2. How to run
.. 3. Tested configurations
.. 4. Output and diagnostics
3. AC-experiments
.. 1. Overview
.. 2. Main scripts
..... 1. Pre-processing
..... 2. OIFS run
.. 3. Configuration
.. 4. Examples
..... 1. Example 1 - A simple run
..... 2. Example 2 - Longer experiment with stop-and-restart
..... 3. Example 3 - Using a workflow script
..... 4. About performance - Long pre-processing
.. 5. Creating ICs at the end of run
..... 1. Example - A series of daily runs
4. Output
5. Using the workflow without restart
━━━━━━━━━━━━━━━━━━━━━━━━━
There are two of directories for the Atmospheric Composition work. The
/t21test_xios_t255ac/ is just a test run with almost no options that you
can set. In /AC-experiments/ you can configure your experiment in
greater detail. In all cases, you need to install some additional data
for the AC work.
1 Test data
═══════════
1.1 Datasets for OIFS/AC
────────────────────────
• You can fetch the main data dedicated to AC configurations from ECFS:
┌────
│ ec:/nm6/oifs/oifs-ac-43r3-chem.tgz
│ ec:/nm6/oifs/oifs-ac-43r3-xtra.tgz
└────
This will unpack to three directories: 43r3/chemistry, 43r3/climate.v015,
and '43r3/xdata'. The second one adds input to the standard
'ifsdata_cy43r3_climate.v015_255l.tgz'
• Unpack them in the same directory where you unpack the default
OpenIFS data, which should then contain:
┌────
│ 43r3/chemistry # AC specific
│ 43r3/xdata # AC specific
│ 43r3/rtables # Standard dataset for OIFS-43r3
│ 43r3/climate.v015 # Standard dataset for OIFS-43r3 + AC specifics
│ 43r3/ifsdata # Standard dataset for OIFS-43r3
│ 43r3/wam # Standard dataset for OIFS-43r3
└────
• A set of initial conditions is available:
┌────
│ ec:/nm6/oifs/oifs-ac-43r3-initcond.tgz
└────
Unpack into a directory: INITIAL-CONDITIONS
• A set of ICMCL climatology for the years 2010 on the TL255L90 grid (kindly
prepared by ECMWF) and few days of 2013 on TL255L60 is also needed:
┌────
│ ec:/nm6/oifs/ICMCL-2010.tgz
└────
Unpack into a directory: ICMCL
1.2 Overview table of provided Initial Conditions
─────────────────────────────────────────────────
The tested and supported configurations are:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
RESOL LEVELS LCHEM CHEM_SCHEME LAERO AERO_SCHEME START_DATE
───────────────────────────────────────────────────────────────────────────────────────────
T255L60-bascoetm5-aer 255 60 true bascoetm5 true aer 2013-07-01
T255L60-tm5-aer 255 60 true tm5 true aer 2013-07-01
───────────────────────────────────────────────────────────────────────────────────────────
T255L91-tm5-aer 255 91 true tm5 true aer 2010-01-01
T255L91-tm5-none 255 91 true tm5 false - 2010-01-01
T255L91-bascoetm5-aer 255 91 true bascoetm5 true aer 2010-01-01
T255L91-bascoetm5-none 255 91 true bascoetm5 false - 2010-01-01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2 t21test_xios_t255ac
═════════════════════
2.1 Presentation
────────────────
• This is as close as possible to /t21test/, so they could eventually
be combined if desired.
• Short 1-day run to demonstrate the OIFS-AC. It has a very limited
set of options. Note that you cannot increase the runtime of the
'oiac' experiment because the provided ICMCL file (albedos/LAIs
climatology) is for 2013-07-01 to 2013-07-03 only.
• The /setup-exp.sh/ script sets the experiment up and run it.
• Check the "Set platform and read its configuration" section, where
the RUNDIR is defined and the OIFS env variables (same as for
compilation) are retrieved.
• You need a wrapper to run it on your HPC (see examples in the
platform subdir)
• XIOS and Chemistry are optional and can be switched on at the
command line. For more details, read script header:
┌────
│ setup-exp.sh -h
└────
• Possible experiments are:
• 'epc8' on the TF21 grid (same as in t21test dir)
• 'oiac' on the TL255 grid, which uses a modified CB05 chemistry
scheme (referred to as TM5) and the AER aerosol scheme on 60
levels. You can switch on the BASCOE chemistry scheme by editing
the CHEM_SCHEME variable in the /setup-exp.sh/.
2.2 How to run
──────────────
• create a wrapper job script that you will submit to your job
manager. The job script must set OMP_NUM_THREADS, NPROC, and
OIFS_RUNCMD (which probably differs if you are using XIOS or not),
and call /setup-exp.sh/. See examples in:
┌────
│ runtime/platform/*job.tmpl
└────
• create a platform configuration file in:
┌────
│ runtime/platform/oifs-config.${platform}.sh
└────
• set the platform variable in /setup-exp.sh/ (line 89)
• submit
2.3 Tested configurations
─────────────────────────
━━━━━━━━━━━━━━━━━━━━━━━━━━━━
cca-cray
────────────────────────────
t21 ok
t21 + xios ok
tm5/aer ok
tm5/aer + xios ok
━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2.4 Output and diagnostics
──────────────────────────
Diagnostics - At the end of 'oiac' experiment, a couple of budgets are
gathered. These can be compared with previously run results, which are
included in the 'ctrl' directory. For example, if you expect the exact
same result, you could:
┌────
│ EXPID=oiac; RUNDIR=${SCRATCH}/${OIFS_CYCLE}/${EXPID} ; RUN_NUMBER=3 #checkoifs
│ fref1=${OIFS_HOME}/t21test_xios_ac/ctrl/massdia_chem__oiac_2013070100.txt
│ fref2=${OIFS_HOME}/t21test_xios_ac/ctrl/ozonbud_chem__oiac_2013070100.txt
│ fnew1=${RUNDIR}/output${RUN_NUMBER}/massdia_chem_oiac.txt
│ fnew2=${RUNDIR}/output${RUN_NUMBER}/ozonbud_chem_oiac.txt
│ diff $fref1 $fnew1
│ diff $fref2 $fnew2
└────
Output:
• ${RUNDIR}/output${RUN_NUMBER} : contains log, chem diagnostics, and
GG+SH grib files if not using XIOS.
• ${RUNDIR}/xios_output : has netcdf files if using XIOS.
For pointers to handle the output on the reduced Gaussian grid, see
the Output section below.
3 AC-experiments
════════════════
3.1 Overview
────────────
In the AC-experiments directory, you will find:
• low level scripts, in the /scripts/ subdirectory. Several of them
provides a set of functions, like a library.
• two main level scripts, one for pre-processing and one for running
OIFS
• one high level workflow script, which is mainly an example, known to
work at ECMWF.
Here's an overview:
┌────
│ AC-experiments
│ ├── config.h # Config file
│ ├── ctrl
│ │ ├── context_ifs.xml # xios resource
│ │ ├── ifs_xml # xios resource
│ │ ├── iodef.xml # xios resource
│ │ ├── namelist.ifs.sh # script to generate OIFS namelist
│ │ └── Table # dir with AC tracers definition tables
│ │
│ ├── oifs-run.sh # Main script to setup rundir and launch OIFS executable
│ ├── prep-ic-icmcl-compo.sh # Main script to gather IC/ICML/ICMCL-COMPO
│ │
│ ├── platform
│ │ ├── ecmwf-cca.job.tmpl # job template to run without xios (PBSpro, qsub)
│ │ ├── ecmwf-cca+xios.job.tmpl # job template to run with xios (PBSpro, qsub)
│ │ ├── ecmwf-cca-workflow.sh # Example workflow script for run with series of restarts
│ │ ├── knmi-rhino.job.tmpl # job template to run without xios (SLURM)
│ │ ├── oifs-config.ecmwf-cca.sh # platform environment (ECMWF)
│ │ └── oifs-config.knmi-rhino.sh # platform environment (KNMI)
│ │
│ └── scripts # Set of low levels scripts
│ ├── add_nrt_fire_chem
│ ├── gaussgr
│ ├── get_alb_lai_intLimits
│ ├── get_tablecol
│ ├── grib_def.h
│ ├── lib_general.sh # lib of utilities
│ ├── lib_chem_setup.sh # lib of functions to create the AC namelists
│ ├── lib_icmcl_clima # lib of functions to create ICMCL files
│ ├── lib_icmcl_compo # lib of functions to create ICMCL-COMPO files
│ └── lib_initcond.sh # lib of functions to create Initial Conditions (IC) files
└────
3.2 Main scripts
────────────────
There are two main scripts to setup and run an experiment. Although
experiments are configured by editing /config.h/, few script arguments
and options allow for a flexible workflow.
3.2.1 Pre-processing
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
The /prep-ic-icmcl-compo.sh/ script prepares the Initial Conditions,
the climatology, and the atmospheric composition fluxes before you run
OIFS. The main purpose is to create or copy three sets of files (IC,
ICMCL and ICMCL-COMPO) needed to run an experiment. Paths to find
those files are set in the /config.h/. See the section DATASETS in
that file for details. Note that it is possible to use different data
than those provide to test OpenIFS/AC but keep in mind this requires
access to MARS archives at ECMWF.
┌────
│ Usage:
│ prep-ic-icmcl-compo.sh START_DATE END_DATE
│
│ Setup or create in $AC_IC_DIR:
│
│ - Initial Conditions for START_DATE (if not continuing/restarting a run)
│ ICs are looked for in $OIFS_INIDATA_DIR. If not found, they are created.
│
│ - ICMCL climatology file for START_DATE to END_DATE.
│ If $ICMCLDIR dir is set (in config.h), the ICMCL file is linked
│ from $ICMCLDIR/ICMCL-yyyymm0100
│ (Note: can contain data for dates before or after those required)
│
│ - ICMCL-COMPO (Emissions and Depositions) for START_DATE to END_DATE.
│ ICMCL-COMPO are looked for in $OIFS_INIDATA_DIR. If not found, it is created.
│
│ (If \$AC_IC_DIR/ICMCL-INIT-COMPO-... exists, it will not be
│ created again. This allows for faster re-run when developing.)
│
│ START_DATE, END_DATE:
│ - Must be in a format understood by 'date -d' (eg 2010-12-15)
│
└────
Note that:
• If prepared beforehand, ICMCL and ICMCL-COMPO can contain data for dates
before or after those requested, but filename formats expected BY THIS
SCRIPT are:
= ICMCL : ICMCL-yyyymm0100
= ICMCL-COMPO : ICMCL-INIT-COMPO or ICMCL-INIT-COMPO-START_DATE-END_DATE (with dates in YYYYMMDDhh)
• $OIFS_INIDATA_DIR, $ICMCLDIR and $AC_IC_DIR are set in /config.h/
3.2.2 OIFS run
╌╌╌╌╌╌╌╌╌╌╌╌╌╌
The /oifs-run.sh/ script reads the experiment setup (config.h), sets a
rundir up (if not already done) and launch the OIFS executable to
start or continue the experiment.
You will need a wrapper around /oifs-run.sh/ to define
OMP_NUM_THREADS, OIFS_NPROC, and OIFS_RUNCMD, and sent it to your HPC
compute nodes. Some example wrappers, working with and without XIOS:
• AC-experiments/platform/ecmwf-cca.job.tmpl
• AC-experiments/platform/ecmwf-cca+xios.job.tmpl
3.3 Configuration
─────────────────
There is a lot of comments in the /config.h/ to explain what the
different entries are. A few words about the main AC keys:
To activate chemistry, set:
• LCHEM=true
• choose chemistry scheme (tm5 or bascoetm5) and corresponding table
version:
• CHEM_SCHEME = tm5 or bascoetm5
• CHEM_VER = ver15 or ver2f
And to activate aerosols:
• LAERO=true
• if not, LCHEM_AEROI (use MACC to get aerosol scattering/absorption)
must be set to false
• choose aerosol scheme and corresponding table version:
• AERO_SCHEME = aer
• AERO_VER = ver1
3.4 Examples
────────────
3.4.1 Example 1 - A simple run
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
• Configure your experiment by editing /config.h/. To run a 10-day
experiment day, use:
┌────
│ run_start_date="2010-01-01"
│ run_end_date="2010-01-11"
└────
The restart frequency should be longer than the run, eg:
┌────
│ rst_freq="1 month"
└────
• Create or gather ICs:
┌────
│ prep-ic-icmcl-compo.sh 2010-01-01 2010-01-11
└────
It is recommended to run it on a compute node, if the ICMCL-COMPO
has to be created.
• Then submit a job to launch an oifs executable.
┌────
│ {qsub, sbatch,...} wrapper.sh
└────
3.4.2 Example 2 - Longer experiment with stop-and-restart
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
• Define your experiment by editing /config.h/. To run a 2-month
experiment with a stop-and-restart after the 1st month (ie 2
legs/chunks of 1 month), just set:
┌────
│ run_start_date="2010-01-01"
│ run_end_date="${run_start_date} + 2 months"
│ rst_freq="1 month"
│ USE_RESTART=true
└────
• Create or gather ICs for every leg:
┌────
│ prep-ic-icmcl-compo.sh 2010-01-01 2010-02-01
│ prep-ic-icmcl-compo.sh 2010-02-01 2010-03-01
└────
These can be run in parallel. But if ICMCLDIR is not provided, then
ICMCL climatology needs to be created, and you have to wait that the
first job is finished before starting the others (which can then run
in parallel).
• Submit your wrapper-around-oifs-run once:
┌────
│ {qsub, sbatch} wrapper.sh
└────
Wait until it is finished, then resubmit it again to simulate the
second month. No need to edit your /config.h/, it will pickup where
it stopped.
3.4.3 Example 3 - Using a workflow script
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
The 4 jobs in the previous example, can be all triggered automatically
and in a timely fashion with a single script. The
/platform/ecmwf-cca-workflow.sh/ provides an example that works on
cca@ECMWF.
• Define a 2-month experiment with a stop-and-restart after the 1st
month:
┌────
│ run_start_date="2010-01-01"
│ run_end_date="${run_start_date} + 2 months"
│ rst_freq="1 month"
└────
• Create and queue jobs with correct dependencies:
┌────
│ ./workflow-cca.sh
└────
3.4.4 About performance - Long pre-processing
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
ICMCL-COMPO could be prepared for the entire run at once. In the
example 2 above, it would be done with:
┌────
│ prep-ic-icmcl-compo.sh 2010-01-01 2010-03-01
└────
Then you simulate the two legs one after the other, like above. There
is no advantage in doing that. Indeed, the creation of the ICMCL-COMPO
can be quite expensive (~15 min for two days!). Better to limit it to
monthly or daily files that can be stitched together afterwards if needed.
3.5 Creating ICs at the end of run
──────────────────────────────────
When setting LAC4IC=true, an AC-for-IC grib file is created at the end
of a run. It is a snapshot of all AC tracers, which can be used by the
pre-processing script to create Initial Conditions. The main
requirement is:
• no netCDF output (LXIOS=false)
With the AC-for-IC file, you can create IC for a new experiment, which
will continue from the previous experiment as far as AC tracers are
concerned. The meteo fields to complete the IC can come from an
archived experiment or a set of standard meteo IC. This workflow is
desirable when you want to reinitialize your integration with an
updated weather forecast (which does not provide AC tracers) for
example.
3.5.1 Example - A series of daily runs
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
• Configure your 1st day experiment:
┌────
│ run_start_date="2010-01-01"
│ run_end_date="2010-01-02"
└────
With a restart frequency equal or longer than the run, you can
switch off the creation of restart:
┌────
│ rst_freq="1 month"
│ USE_RESTART=false
└────
• Create ICs
┌────
│ prep-ic-icmcl-compo.sh 2010-01-01 2010-01-02
└────
• Then submit a job to launch oifs executable
┌────
│ {qsub, sbatch} wrapper.sh
└────
This will create an extra file we will use in the next day
experiment:
┌────
│ AUXFILE=${RUNDIR}/output/001/ICMUA${EXP}.ac_rst
└────
• Configure your next day experiment, with a new name (or after
removing the previous rundir - after moving away the AUXFILE!):
┌────
│ EXP=bcde
│ run_start_date="2010-01-02"
│ run_end_date="2010-01-03"
└────
In the /config.h/, be sure to:
- set OIFS_INIDATA_DIR to nothing, or comment it.
- set the variable AC_IC_ONCE to the AUXFILE from the previous exp.
- set the MET_IC_DIR or modify the "variables to retrieve METEO initial
conditions from an archived experiment" section as you see fit.
• Create ICs
┌────
│ prep-ic-icmcl-compo.sh 2010-01-02 2010-01-03
└────
• Then submit a job to launch oifs executable
┌────
│ {qsub, sbatch} wrapper.sh
└────
This will create an extra file for another run:
┌────
│ AUXFILE=${RUNDIR}/output/001/ICMUA${EXP}.ac_rst
└────
• Etc.
4 Output
════════
Output data are on the reduced grid - which is an unstructured grid in the
netcdf files. To go further, you may want to remap the output. This can be
achieved with /cdo/. The procedure is slightly different for grib and netCDF
files. Check the scripts in the /postproc/ directory for examples dealing
with grib files. For netCDF files, you must set a grid (grid descriptions
are available in the /ctrl/ directory) before remapping:
┌────
│ cdo -L setgridtype,regular -setgrid,${OIFS_HOME}/t21test_xios_ac/ctrl/TL255-griddes.txt \
│ AMIP_6h_reduced_ml.nc oiac_6h_reduced_ml.nc
└────
Note that available UV-processor output is an experimental product, which is
currently not filled (just zeroes). It concerns two variables: uvp2drad
(gribcode 210055) and toaer340 (gribcode 210056)