Note |
---|
Note that this documentation applies to the OpenIFS/AC code extension for OpenIFS 43r3. This does not apply to users of OpenIFS 48r1 or later model releases. |
...
Table of Contents |
---|
...
This page describes how OpenIFS/AC v1 (derived from OpenIFS 43r3v2) can be obtained, built and used for forecast experiments. The documentation shows examples of how to use the model on the ECMWF Atos Sequana XH2000 HPCF (hpc2020). Adjustments will need to be made for local installations.
Info |
---|
Enquiries about OpenIFS/AC or technical questions should in the first instance be sent to openifs-support@ecmwf.int |
OpenIFS/AC compared to the standard OpenIFS model
OpenIFS/AC consists of the standard OpenIFS model and the
Table of Contents |
---|
This page describes the required steps to use OpenIFS/AC "v2ac-rc" on the ECMWF HPCF.
OpenIFS/AC compared to the standard OpenIFS model
OpenIFS/AC consists of the standard OpenIFS model and the additional composition software package. Both components together are referred to as OpenIFS/AC (with AC standing for "atmospheric composition").
The standard OpenIFS model is described in detail in the OpenIFS 43r3 User Guide. The user guide introduces the model's components and supported model grids. It informs about additional required software packages and how to build the OpenIFS model. It further describes how to carry out an acceptance test after a successful model build in order to verify the model's functionality.
Info |
---|
Users who are new to OpenIFS should first read the OpenIFS 43r3 User Guide's sections 1 to 5 before proceeding with OpenIFS/AC, as the steps described therein are required for building OpenIFS/AC. The instructions in the user guide will not be repeated here. |
...
In the standard OpenIFS model the source code for atmospheric composition (atmospheric chemistry and aerosol code) has been replaced by dummy routines. The additional model sources for OpenIFS/AC replace these dummy routines and substitute the full composition source code. It is therefore necessary to rebuild the binary model executable after adding the additional model source code. This process is described further below.
...
In order to run OpenIFS with atmospheric composition a substantial number of additional input data files are required. These provide initial conditions for the chemical and aerosol fields as well as additional boundary conditions for the model, mostly in the form of emission fluxes for a number of chemical tracers. These emissions are released mostly at the model surface, however they can also be associated with emission heights (e.g. for engine emissions from global air traffic). Further information is required for the physical removal of tracers through either wet scavenging or dry deposition, the latter of which is related to the surface vegetation type as used by the land surface scheme.
The additional data packages required for atmospheric composition modelling need to be prepared for the model at the desired grid resolution. While the standard OpenIFS model can be used with a wide range of model grids, the prepared chemical input data is only available for few selected grid resolutions and for specific time periods. If the model is to be used for other time periods or at different horizontal or vertical grid resolutions then the additional data packages need first to be generated for this purpose.
Info |
---|
Note that the additional data packages required for OpenIFS/AC are only made available for specific time periods and grid resolutions. |
...
Obtaining
...
the model code
This section contains information about where the source code package for OpenIFS/AC can be obtained and how the sources should be installed.
Working Working with the ftp server
In order to convert the standard OpenIFS model to OpenIFS/AC the additional software package for composition needs to be installed into an existing OpenIFS installation.
If both standard (meteorological) model and the atmospheric composition model are to be used at the same time, it is recommended to create a new OIFS_HOME directory for OpenIFS/AC and to first unpack the standard OpenIFS model sources into this directory (for instance, if the standard OpenIFS model is installed into directory 'oifs43r3', the new destination for the model sources could be extracted into directory 'openifs-ac').
Once the standard OpenIFS model code is unpacked, unpack package <package-name> into $OIFS_HOME, overwriting any pre-existing files from the standard model installation.
TO DO: we need a separate code package for the OpenIFS/AC code that can be unpacked into the existing OpenIFS 43r3v2 source code distribution
Working with the git repository
OpenIFS/AC is maintained in Philippe Le Sager's ECMWF git repository: https://git.ecmwf.int/users/nm6
No Format |
---|
git clone ssh://git@git.ecmwf.int/~nm6/oifs43r3.git
mv oifs43r3 openifs-ac
cd openifs-ac
git checkout v2ac-rc |
Environment settings
In order to build OpenIFS/AC the standard environment settings for oifs43r3v2 can be used with the application of two changes:
- change OIFS_HOME to point to the installation of openifs-ac (and not to the standard oifs43r3 installation)
- change OIFS_DATA_DIR to a bespoke path as the additional data packages will be added to the climate data files; we describe below what date needs to be moved to this directory
- before compiling the model sources edit $OIFS_HOME/make/oifs.fcm to enable chemistry (which is by default disabled)
It is recommend for the first two changes to use a custom configuration script to set the environment variables.
Alternatively the settings can be enabled when running the prep scripts that come with OpenIFS/AC: For the t21 acceptance tests, the environment variables and paths are set within the oifs-config.<my_name>.sh which needs to be edited in the platform
subdirectory. This is described below.
Obtaining additional data packages
...
Download the additional data tarballs that are required for chemistry experiments into /data/openifs-ac :
No Format |
---|
cd /data/openifs-ac
module load ecfs
ecp ec:/${datapath}/oifs-ac-43r3-chem.tgz .
ecp ec:/${datapath}/oifs-ac-43r3-xtra.tgz .
ecp ec:/${datapath}/oifs-ac-43r3-initcond.tgz .
ecp ec:/${datapath}/ICMCL-2010.tgz . |
...
AC specific
...
You need to connect to the ECMWF ftp server to download the source package for OpenIFS/AC. If your institution has signed the OpenIFS license agreement then you can request the ftp password by emailing openifs-support@ecmwf.int.
You can either convert your existing OpenIFS installation or create a new installation for OpenIFS/AC. We recommend to create a new installation and for this you will also need the sources of the standard OpenIFS model (oifs43r3v2) which can also be found on the ftp server.
How to establish a connection to the ECMWF ftp server and how to obtain OpenIFS 43r3 sources is described in more detail here in the OpenIFS User Guide.
No Format |
---|
ftp ftp.ecmwf.int
(login with username and password for OpenIFS users)
cd src/openifs-ac
get openifs-ac.43r3.v1.gz
bye |
Installing the OpenIFS/AC sources
Installing the model sources and building the model (including acceptance tests) requires approximately 3-4 GB of disk space. The additionally required climatological data packages are not yet included; their installation is described below and will require further disk space.
Method 1: We recommend to create a new installation for OpenIFS/AC. This will preserve your original OpenIFS installation and will allow you to use the standard (meteorological) model and the atmospheric composition model interchangeably.
To proceed you should:
- Create a new home for the OpenIFS/AC installation. Here we call this location /home/openifs-ac.
- Change into your new directory and unpack therein the original OpenIFS model sources: tar xvzf oifs43r3v2.tgz
- In a second step, in the same folder gunzip the OpenIFS/AC sources and patch the original OpenIFS:
- gunzip openifs-ac.43r3.v1.gz
- patch -p1 <openifs-ac.43r3.v1
Info |
---|
Note that some files of the original model will be modified. It is therefore essential to install the original OpenIFS 43r3v2 sources first before installing the OpenIFS/AC sources. |
Method 2: Alternatively, in order to convert the standard OpenIFS model to OpenIFS/AC the additional software package for composition needs to be installed into an existing OpenIFS installation. You can create separate binary executables, with and without chemistry enabled. We recommend to create first a backup of your original installation before proceeding.
For the conversion you should:
- Change into your $OIFS_HOME directory where OpenIFS is installed: cd $OIFS_HOME
- In this directory extract the downloaded source package:
- gunzip openifs-ac.43r3.v1.gz
- patch -p1 <openifs-ac.43r3.v1
Info |
---|
Note that this unpacking will not only add new files to your existing $OIFS_HOME but it will also overwrite some of the existing files with updated versions required for OpenIFS/AC. |
Obtaining additional data packages
Anchor | ||||
---|---|---|---|---|
|
In addition to the enhancements of the model sources, OpenIFS/AC also requires additional external data to carry out forecast experiments (e.g. climatological data for boundary conditions, fluxes for emitted chemical compounds, etc). This data will complement the data found usually in the $OIFS_DATA_DIR location. The data volume for the standard distribution of OpenIFS/AC is approximately 21 GB.
Info |
---|
Note: It is recommended to create a new OIFS_DATA_DIR directory specifically for use with OpenIFS/AC which is kept separate from the data directory for the meteorological model. |
As a next step you should:
- Create a new directory which will contain the additional data that is required for atmospheric composition modelling and it will also contain the new
OIFS_DATA_DIR
. Here we call this location /data/openifs-ac. Download the required four additional data tgz archives into /data/openifs-ac. The following commands will work on the Atos HPC:
No Format cd /data/openifs-ac module load ecfs ecp ec:/nm6/oifs/oifs-ac-43r3-chem.tgz . ecp ec:/nm6/oifs/oifs-ac-43r3-xtra.tgz . ecp ec:/nm6/oifs/oifs-ac-43r3-initcond.tgz . ecp ec:/nm6/oifs/ICMCL-2010.tgz .
Unpack oifs-ac-43r3-chem.tgz and oifs-ac-43r3-xtra.tgz into /data/openifs-ac. This will result in the following directory structure inside /data/openifs-ac :
43r3/chemistry
AC specific 43r3/xdata
AC specific 43r3/rtables
Standard dataset for OpenIFS 43r3 43r3/climate.v015
Standard dataset for OpenIFS 43r3 + AC specifics 43r3/ifsdata
Standard dataset for OpenIFS 43r3 43r3/wam
Wave model data for OpenIFS 43r3, needs to be added explicitly
The following two tgz archives can be unpacked to any location, however in this example the files we will also extract them into /data/openifs-ac:
- Unpack oifs-ac-43r3-initcond.tgz into /data/openifs-ac which results in a directory /data/openifs-ac/INITIAL-CONDITIONS.
The files therein are initial data files for the first day of month for all 12 months of 2010. - Unpack ICMCL-2010.tgz into /data/openifs-ac which results in a directory /data/openifs-ac/ICMCL.
This directory contains monthly files with surface boundary conditions (e.g. SSTs, soil temps, albedos, LAI etc). - In the tgz archives above, the data for the year 2013, provided on a Tl255L60 grid, is required for the T21 acceptance tests. The data for year 2010 is provided on a Tl255L91 grid and can be used for the example forecasts.
- From your conventional OpenIFS installation's
OIFS_DATA_DIR
copy the directories ifsdata and rtables to /data/openifs-ac/43r3 as they are also needed here. If this is your first model installation, these data directories are also available from the ftp server. - Further, create a directory /data/openifs-ac/43r3/wam which needs to contain wave model initial data for 2010-01-01.
Info |
---|
Note: In the reference forecast experiments discussed below the initial conditions files for the wave model (ocean surface waves) are added to the experiment separately from a central location in the |
...
Building the model
This section describes how to build the binary model executable for OpenIFS/AC. Most problems related to building the model executable are due to incorrect environment settings. Therefore we recommend that particular care should be given to the environment settings to ensure that they match the local hardware platform.
Environment settings
Similar to the standard OpenIFS model, OpenIFS/AC requires system and user software libraries (e.g. eccodes, netcdf, etc) and relies on shell environment variables which need to be set according to your local installation. In order to avoid the duplication of documentation we refer to the OpenIFS User Guide as the prerequisites for OpenIFS are also valid for OpenIFS/AC. For more information about environment variable settings please see here.
Info |
---|
We recommend to start from a successfully tested local environment for OpenIFS and work from there to make the necessary adjustments for OpenIFS/AC. |
In order to build OpenIFS/AC the standard environment settings for oifs43r3v2 can be used with the application of two changes:
- Change $OIFS_HOME to point to your installation of openifs-ac (and not to the standard oifs43r3 installation), in our example this is /home/openifs-ac.
- Change $OIFS_DATA_DIR to the location with the additional data packages, in our example this is /data/openifs-ac.
- Before compiling the model sources edit $OIFS_HOME/make/oifs.fcm to enable chemistry (which is by default disabled).
It is recommend to apply the first two changes in $OIFS_HOME/oifs-config.editme.sh and to activate the settings with the command: source ./oifs-config.editme.sh
before attempting to build the model.
In order to run OpenIFS/AC a number of further changes and additions to the environment are required. These are described below.
Building the binary executable
On the ECMWF HPCF it is possible to compile the model using a login node. However, in order to complete the acceptance testing with chemistry an interactive node should be requested.
Code Block | ||||
---|---|---|---|---|
| ||||
% ecinteractive -c 12 -m 32GB -t 06:00:00 |
If you have a configuration script that sets the appropriate environment for OpenIFS you should run this first before building the model executables.
In your command line you should type
No Format |
---|
export OIFS_CHEM=enable |
to overwrite the default setting in oifs.fcm (or alternatively you can edit the variable in file oifs.fcm)
Further you should make a choice whether you want to build with XIOS server support, in that case set appropriate environment variables. Otherwise set
No Format |
---|
export OIFS_XIOS=disable |
to disable XIOS.
Then compile the model using fcm (here with verbose output and with 16 parallel child processes):
No Format |
---|
cd $OIFS_HOME/make
fcm make -v -j 16 -f oifs.fcm |
You should only proceed further after the code has compiled and all the executables were built without errors.
...
Running the acceptance tests
The model package offers various tests to check the basic functionality of the model and to verify that all paths to the data directories are correct and accessible.
Overview
The acceptance tests are run on a T21 full Gaussian grid and are designed to require a minimum of computational resources. Several different tests are being offered which activate different pathways in the model source code. It is highly recommended to carry out these brief tests to verify that the model is working normally prior to using the model for forecast experiments.
The tests are located in $OIFS_HOME/t21test
and t21test_xios_t255ac
.
Running the tests for the first time requires editing the wrapper scripts job.sh
and setup-exp.sh
by setting the path to the appropriate master.exe
and by changing RUNCMD to to something appropriate for the platform.
t21test: This directory contains the original acceptance test which is provided with OpenIFS. It should work in the same way for OpenIFS/AC as described in the OpenIFS User Guide.
t21test_xios: This directory contains an OpenIFS acceptance test which makes use of the XIOS server – we do not describe this here.
t21test_xios_t255ac: This directory contains the new tests for OpenIFS/AC. The test consists of a short 1-day run to demonstrate the working of OpenIFS/AC. It has a very limited set of options. Note that you cannot increase the runtime of the experiment because the provided ICMCL file does not contain data for an extended model run. The three acceptance tests in this directory make use of the additional compositon (AC) code and it is possible to run this test (a) without AC as in t21test, (b) with additional AC included, and (c) for both of the previous tests additionally XIOS server output can be selected.
Note |
---|
Note: If the XIOS server is to be used in these tests then a precompiled version of its libraries needs to be accessible and the appropriate paths need to be set in oifs-config.{platform}.sh. We do not describe here how XIOS is built, more information on XIOS can be found here. Note: During the acceptance test with enabled chemistry the following error messages from the Rosenbrock solver can occur at times. In the t21test a small number of occurrences of this error can be ignored:
|
Step-by-step
Here we describe the required steps to carry out the tests.
Using t21test
No Format |
---|
./job.sh -n 12 -t 2 -m ../make/gnu-opt-chem-fftw/oifs/bin/master.exe |
If the oifs_run script (or the oiac_run script) is used without the --nomove option it produces an error at the end of the acceptance test, stating the ICMGGepc8+* files cannot be found. This refers to the copying of model output to output1 and the postprocessing and is therefore not relevant to the model functionality.
Test result: This test runs in directory $OIFS_HOME/t21test
and model output is written into subfolder output1
. The model run also produces the file res_021_0072
which contains results from the error calculation. This error should be below 5%. For more information see the OpenIFS User Guide.
Using t21test_xios_t255ac
Without chemistry: This test uses a different script compared to the standard OpenIFS model's t21test. First we will verify its correct functionality by running the test without activating the model chemistry.
- To do this we run the standard t21test for experiment epc8 (T21L19) with the
setup-exp.sh
new script. - The name for the hardware platform needs to be set (l.89, e.g.
platform=ecmwf-hpc2020
) - The test experiment runs in the
RUNDIR
directory specified insetup-exp.sh
(l.94, the default is$SCRATCH/43r3/epc8/
). - We also need to use the command line option -x and give as argument the absolute full path to the
master.exe
binary executable. - In
setup-exp.sh
you should change the value forRUN_NUMBER
otherwise any pre-existing model output from this test will be overwritten.
No Format |
---|
./setup-exp.sh -n 16 -t 2 -x /home/damk/oifs/model/openifs-ac/make/gnu-opt-chem-fftw/oifs/bin/master.exe |
Test result: As before an error calculation file res_021_0072
is created in the run directory and the model output is found in the subfolder output1
(this name depends on the RUN_NUMBER variable).
Info |
---|
Running on the ECMWF HPCF: |
With chemistry: Now repeat the test with enabled model chemistry. This is a short 1-day test run that uses the newly added source code of OpenIFS/AC. It has a very limited set of options and should not be used for forecast experiments. Note that you cannot increase the experiment duration beyond one day because the boundary conditions are only supplied from 2013-07-01 to 2013-07-03.
This experiment is named oiac
and uses a Tl255L60 grid and runs for 24 hours. The default run directory is $SCRATCH/43r3/oiac/
.
No Format |
---|
./setup-exp.sh -c -n 16 -t 2 -x /home/damk/oifs/model/openifs-ac/make/gnu-opt-chem-fftw/oifs/bin/master.exe |
This test runs with a 30 min time step and due to the computational requirements for the added chemistry it will take much longer to complete. The variable CHEM_SCHEME offers the choice of two chemistry schemes, the AER aerosol scheme is always enabled.
Note |
---|
Note: If the XIOS server is to be used in the t21test experiments (option |
Test result: This experiment calculates budgets of atmospheric mass, along with overall production and loss tendencies, as well as more detailed reaction budget terms relevant for the ozone production and loss tendency. These are printed in the NODE.001_01
file and afterwards extracted to the files $RUNDIR/output{$RUN_NUMBER}/massdia_chem_oiac.txt
and ozonbud_chem_dia.txt
.
These diagnostics can be compared with pre-calculated results in files massdia_chem__oiac_2013070100.txt
and ozonbud_chem__oiac_2013070100.txt
which are located in $OIFS_HOME/t21test_xois_t255ac/ctrl/
. For example this could be done with a small shell script such as illustrated in the box below.
No Format |
---|
EXPID=oiac; RUNDIR=${SCRATCH}/${OIFS_CYCLE}/${EXPID} ; RUN_NUMBER=3 #checkoifs
fref1=${OIFS_HOME}/t21test_xios_ac/ctrl/massdia_chem__oiac_2013070100.txt
fref2=${OIFS_HOME}/t21test_xios_ac/ctrl/ozonbud_chem__oiac_2013070100.txt
fnew1=${RUNDIR}/output${RUN_NUMBER}/massdia_chem_oiac.txt
fnew2=${RUNDIR}/output${RUN_NUMBER}/ozonbud_chem_oiac.txt
diff $fref1 $fnew1
diff $fref2 $fnew2 |
As a rule of thumb, differences between the reference budget numbers and those from the t21test should be only small (below 5%), although this depends on the exact variable that is assessed.
...
Single forecast experiments
This section describes the required steps to carry out a single forecast experiment (no restarts) with chemistry. The essential three steps can be summarised as:
- Edit paths and variables in
$OIFS_HOME/AC-experiments/config.h
. This controls model switches and the chemistry scheme. - Prepare initial conditions using
$OIFS_HOME/AC-epxeriments/prep-ic-icmcl-compo.sh
. - Customise a job wrapper script for the relevant hardware platform (a
*.job
file) in order to submit theoifs-run.sh
script to the batch scheduler.
Overview of the AC-experiments directory
No Format |
---|
AC-experiments/
├── config.h # MAIN experiment config file
├── ctrl/
│ ├── context_ifs.xml # xios resource
│ ├── ifs_xml/ # xios resource
│ ├── iodef.xml # xios resource
│ ├── namelistfc
│ ├── namelist.ifs-epc8.sh # namelist for t21test without chemistry
│ ├── namelist.ifs-oiac.sh # namelist for t21test with chemistry
│ ├── namelist.ifs.sh # namelist template for forecast experiments
│ ├── Table/ # dir with AC tracers
│ └── wam_namelist
├── ecmwf-hpc2020.job # job wrapper script for oifs-run.sh
├── oifs-run.sh # MAIN run script
├── platform/ # dir with platform-specific templates
├── prep_emis/ # dir with emission utilities
├── prep-ic-icmcl-compo.sh # MAIN script to prepare initial/boundary conditions
└── scripts/ # dir with low-level scripts
├── add_nrt_fire_chem
├── gaussgr
├── get_alb_lai_intLimits
├── get_tablecol
├── grib_def.h
├── lib_chem_setup.sh
├── lib_general.sh # lib of utilities
├── lib_icmcl_clima.sh # lib of functs to create ICMCL boundary cond. files
├── lib_icmcl_compo.sh # lib of functs to create ICMCL-COMPO files
├── lib_initcond_offline.sh
└── lib_initcond.sh # lib of functs to create initial conditions files |
The experiment controls are in $OIFS_HOME/AC-experiments/.
There the following essential files can be found:
- prep-ic-icmcl-compo.sh - a script to generate initial conditions for chemical fields for the selected run time. Begin and end date need to be given as an argument.
- config.h - settings for model installation, paths, and experiment configuration.
- oifs-run.sh - this script controls the workflow for the experiment: reads experiment config, sets up run directories (in $SCRATCH), loads platform configuration, edits namelists, links to initial conditions and executes OIFS_RUNCMD
- ecmwf-hpc2020.job wrapper script for oifs-run.sh, required settings for the batch scheduler
In $OIFS_HOME/AC-experiments/platform:
- it is necessary to create a platform-specific machine config file: oifs-config.ecmwf-hpc2020.sh
The above ecmwf-hpcs2020.job wrapper script to submit the batch job is based on templates in the platform directory; a number of templates exist and for the ECMWF HPCF the following templates are relevant:
- ecmwf-hpc2020.job.tmpl – run oifs without XIOS
- oifs-config.ecmwf-hpc2020.sh – template for the machine config file
Editing the experiment configuration file
For the forecast experiment to succeed and in order to generate the initial data correctly, you will need to edit variables inside the file config.h
to correspond with the chosen settings for your experiment.
In config.h you should edit the following variables. The example is for a single forecast (without restarts or multiple forecast legs), starting from 1st January 2010 having 48 hours duration.
No Format |
---|
PLATFORM='ecmwf-hpc2020' # this needs to match the name of the platform configuration file
EXP=abcd # you should select here an experiment ID (4-letter alphanumeric word, must not start with '0')
RUNDIR=${SCRATCH}/43r3/experiments/${EXP}/runs # this is the experiment directory
AC_IC_DIR=${SCRATCH}/43r3/experiments/${EXP}/chem_ic # this is the staging directory for initial and boundary conditions data files
# Simulation start and end date. If setting time (not mandatory), **put it before** the date
run_start_date="00:00 2010-01-01"
run_end_date="${run_start_date} + 2 days"
rst_freq='6 day' # Run for 6 days without restarts. Our experiment will finish after 2 days, so this will be ignored.
USE_RESTART=false # Do not write restart files. |
Path variables need to be set to the appropriate locations. In our example described in the Section above these locations can be found in the OIFS_DATA_DIR
under /data/openifs-ac.
No Format |
---|
OIFS_INIDATA_DIR=${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-tm5-aer # initial and boundary conditions for the chemistry scheme
ICMCLDIR=${OIFS_DATA_DIR}/ICMCL # physical/meteorological boundary conditions
MET_IC_DIR=${OIFS_DATA_DIR}/ICMCL/aabg/2010010100 # path for meteorological initial data in case it needs to be generated
AC_IC_ONCE=${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-tm5-aer/ICMUAoiac.ac_rst # initial conditions for chemical tracers |
Further variables and settings that need changing in config.h
are described below in the next section for four possible forecast experiment configurations.
Once all the changes have been made in config.h, the experiment initial and boundary condition files can be prepared as described in the following section.
After the data has been prepared the platform-specific job wrapper script can be executed to submit the model experiment job to the batch scheduler.
Example configuration settings for single forecast experiments
Select one of the following four T255L91 forecast configurations for your experiment. The tested and supported configurations are:
Experiment | RESOL | LEVELS | LCHEM | CHEM_SCHEME | LAERO | AERO_SCHEME | START_DATE | |
---|---|---|---|---|---|---|---|---|
T255L60-bascoetm5-aer | t21test | 255 | 60 | true | bascoetm5 | true | aer | 2013-07-01 |
T255L60-tm5-aer | t21test | 255 | 60 | true | tm5 | true | aer | 2013-07-01 |
T255L91-tm5-aer | forecast | 255 | 91 | true | tm5 | true | aer | 2010-01-01 |
T255L91-tm5-none | forecast | 255 | 91 | true | tm5 | false | -- | 2010-01-01 |
T255L91-bascoetm5-aer | forecast | 255 | 91 | true | bascoetm5 | true | aer | 2010-01-01 |
T255L91-bascoetm5-none | forecast | 255 | 91 | true | bascoetm5 | false | -- | 2010-01-01 |
Using TM5 chemistry without aerosol
Example: Run a 2-day forecast starting from 2010-01-01 with TM5 chemistry without aerosol.
Use the following settings in config.h:
Code Block |
---|
PLATFORM='ecmwf-hpc2020'
RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs' # where your experiment will run
AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic' # where your chemical initial conditions will be created
run_end_date="2010-01-03"
rst_freq='6 day'
USE_RESTART=false
LCHEM=true
CHEM_SCHEME="tm5"
CHEM_VER="ver15"
OIFS_INIDATA_DIR='${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-tm5-none'
LAERO=false
LAERCHEM=false
LCHEM_AEROI=false
LINJ_CHEM=false
LINJ_AER=false
LAEROSFC=false
LWAM=true
LRLXG=false |
Using TM5 chemistry with AER aerosol
Example: Run a 2-day forecast starting from 2010-01-01 with TM5 chemistry and the AER bulk aerosol scheme.
Use the following settings in config.h:
Code Block |
---|
PLATFORM='ecmwf-hpc2020'
RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs' # where your experiment will run
AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic' # where your chemical initial conditions will be created
run_end_date="2010-01-03"
rst_freq='6 day'
USE_RESTART=false
LCHEM=true
CHEM_SCHEME="tm5"
CHEM_VER="ver15"
OIFS_INIDATA_DIR='${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-tm5-aer'
LAERO=true
LAERCHEM=true
LCHEM_AEROI=true
LINJ_CHEM=true
LINJ_AER=true
LAEROSFC=true
LWAM=true
LRLXG=false |
Using BASCOE-TM5 chemistry without aerosol
Example: Run a-2 day forecast starting from 2010-01-01 with combined BASCOE and TM5 chemistry without aerosol.
Use the following settings in config.h:
Code Block |
---|
PLATFORM='ecmwf-hpc2020'
RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs' # where your experiment will run
AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic' # where your chemical initial conditions will be created
run_end_date="2010-01-03"
rst_freq='6 day'
USE_RESTART=false
LCHEM=true
CHEM_SCHEME="bascoetm5"
CHEM_VER="ver2f"
OIFS_INIDATA_DIR='${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-bascoetm5-none'
LAERO=false
LAERCHEM=false
LCHEM_AEROI=false
LINJ_CHEM=false
LINJ_AER=false
LAEROSFC=false
LWAM=true
LRLXG=false |
Using BASCOE-TM5 chemistry with AER aerosol
Example: Run a-2 day forecast starting from 2010-01-01 with combined BASCOE and TM5 chemistry without aerosol.
Use the following settings in config.h:
Code Block |
---|
PLATFORM='ecmwf-hpc2020'
RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs' # where your experiment will run
AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic' # where your chemical initial conditions will be created
run_end_date="2010-01-03"
rst_freq='6 day'
USE_RESTART=false
LCHEM=true
CHEM_SCHEME="bascoetm5"
CHEM_VER="ver2f"
OIFS_INIDATA_DIR='${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-bascoetm5-none'
LAERO=true
LAERCHEM=true
LCHEM_AEROI=true
LINJ_CHEM=true
LINJ_AER=true
LAEROSFC=true
LWAM=true
LRLXG=false |
Generating initial data
Initial conditions for the standard OpenIFS model can be generated using the OpenIFS Data Hub or alternatively they can be requested from the ECMWF OpenIFS support by email openifs-support@ecmwf.int. This provides initial experiment data and surface boundary conditions for the physical/meteorological NWP forecast model.
In order to simulate atmospheric composition with OpenIFS/AC additional initial and boundary conditions are needed. These can be created using the prep-ic-icmcl-compo.sh
script.
Info |
---|
Note: The prep-ic-icmcl-compo.sh script requires external software: the CDO command line operators to manipulate and analyse climate and NWP model data which are available here: https://code.mpimet.mpg.de/projects/cdo/. If these utilities are not available on your computer system then they need to be installed before using the script. |
The prep-ic-icmcl-compo.sh script (henceforth: prep script) carries out 4 tasks:
- Creates a destination directory $AC_IC_DIR which serves as a staging area for all initial and boundary conditions required for the forecast experiment. Symbolic links will point later from inside the experiment run directory to these files.
- Copies the physical/meteorological initial conditions files that must be specific to the experiment start date from $OIFS_INIDATA_DIR to the new $AC_IC_DIR. If these files cannot be found then an attempt is made to create them from files found in the $MET_IC_DIR directory.
- Copies the physical boundary conditions file ICMCL{exp}INIT (SSTs, albedos, etc) from $OIFS_INIDATA_DIR to $AC_IC_DIR and if necessary truncate to length of experiment duration.
- Copies the chemistry initial and boundary conditions file (emission fluxes, deposition velocities) ICMCL-INIT-COMPO from $OIFS_INIDATA_DIR to $AC_IC_DIR. If this file cannot be found it will be created.
The syntax to run the prep script is:
No Format |
---|
./prep-ic-icmcl-compo.sh <start_date> <end_date> |
with start_date and end_date having the format YYYY-MM-DD.
In the examples above for a 2-day forecast, the syntax should be ./prep-ic-icmcl-compo.sh 2010-01-01 2010-01-03
to generate the data for 48 hours from 1st January 2010.
Running the prep script will take a few minutes and will result in new data that will be written to $AC_IC_DIR
, by default this location is $SCRATCH/43r3/experiments/{exp}/chem_ic
.
This folder serves as a staging area for the initial and boundary conditions data and if the target files in $AC_IC_DIR
already exist then they they will not be overwritten.
Creating initial conditions at the end of an experiment run
When setting LAC4IC=true
an AC-for-IC grib file is created at the end of a run. It is a snapshot of all chemical/atmospheric composition tracers. This file can be used by the pre-processing script (prep script) to create new initial conditions.
With the AC-for-IC file, initial conditions for a new experiment can be generated which will provide a continuation of the chemical evolution of the tracer fields from the previous experiment. The meteorological fields to complement the initial conditions can originate from an archived experiment or can be generated with the OpenIFS Data Hub. This workflow is desirable when aiming to reinitialise the model integration with an updated weather forecast or analysis data (which does not provide AC tracers).
In order to restart with atmospheric composition tracers from the AC-for-IC grib file select the following settings in config.h
:
- Either leave the variable
OIFS_INIDATA_DIR
empty, delete it or comment it out. - Set the variable
AC_IC_ONCE
to point to the generated AC-for-IC grib file from the previous experiment. This file is typically found in this location in the previous experiment:${RUNDIR}/output/${RUN_NUMBER}/ICMUA${EXP}.ac_rst
- Set the variable
MET_IC_DIR
to the relevant path for the new experiment's initial conditions or modify the "variables to retrieve METEO initial conditions from an archived experiment" section as appropriate.
Running the forecast experiment
Now the actual forecast experiment can be submitted to the batch scheduler using the platform-specific job wrapper script. Examples for this are found in $OIFS_HOME/AC-experiments/platform
. An example template for the ECMWF HPCF Atos Sequana XH2000 is the file ecmwf-hpc2020.job.tmpl
.
The actual forecast experiment will be carried out inside the run directory which is defined in config.h
with variable RUNDIR
. The default on the ECMWF HPCF is $SCRATCH/43r3/experiments/${exp}/runs/
.
Due to the much increased computational requirements for simulating atmospheric composition, forecast experiments are often broken into individual parts (sometimes referred to as "legs"). For each leg a restart of the forecast experiment needs to be carried out.
Model Output
The oifs-run.sh
script for OpenIFS/AC has been designed to be able to cope with restarts or "legs", and the model output from each experiment leg is stored in individual subfolders named by leg number. In the case of a single forecast experiment without restarts the output will be written to $RUNDIR/output/001
.
Log files
Log file and experiment-related configuration data (namelist, NODE file, ifs.stat), as well as text files with diagnostic model outuput (e.g. mass and ozone budgets) are copied to $RUNDIR/log/001
.
Restart Data
Model data relevant for restarts (not discussed here on this page) will be stored in $RUNDIR/restart/${next_leg_number}.
XIOS output
Files relevant to XIOS server output are stored in $RUNDIR/ifs_xml (not discussed here on this page).
...
Multiple forecasts with restarts
Info |
---|
Note: This option is at an experimental development stage and it is therefore currently not supported for general users. Its implementation depends on scripting that is dependant on the hardware platform. Suggestions for script templates that are used on the ECMWF HPCF can be provided on request. |
There are several reasons why a user may wish to restart a forecast experiment. The increase in computational costs for simulating atmospheric composition may require to break a longer forecast experiment into individual sections ("legs"). Also, a restart will allow to initialise the meteorology in the model with data taken from an analysis of the current modelled time.
Due to the time scales associated with the different chemical processes and the lifetime of chemical tracers with respect to their loss processes, it is often convenient to retain the chemical tracer fields in the model simulation and update only the driving meteorology at the time of the restart. In some applications, the model could even be initialised every single day with an up-to-date meteorological analysis field while the chemical fields are left to evolve. The previous section described how chemical initial conditions can be created at the end of an experiment run, thus retaining the chemical tracer information from the previous experiment leg.
We will here provide some examples of the principle steps required for restarting experiments.
Example 1: Longer experiment with manual stop and restart
The experiment parameters need to be defined by editing config.h
. The following settings are used for a 2-month experiment with a stop-and-restart after the first month. This will result in two experiment legs of 1 month duration each.
No Format |
---|
run_start_date="2010-01-01"
run_end_date="${run_start_date} + 2 months"
rst_freq="1 month"
USE_RESTART=true |
The initial conditions for each leg need to be either created or gathered from a prepared repository.
No Format |
---|
prep-ic-icmcl-compo.sh 2010-01-01 2010-02-01
prep-ic-icmcl-compo.sh 2010-02-01 2010-03-01 |
These processes can be run simultaneously. The meteorological fields and initial conditions can be requested from the OpenIFS Data Hub. The wrapper script needs to be submitted for the first experiment leg. Once the experiment has completed, config.h needs to be edited to use the initial experiment data from the date and time when the first leg has stopped. Then the wrapper script for the second month should be submitted.
Example 2: Using a workflow script
The 4 jobs in the previous example, can be all triggered automatically and in a timely fashion with a single script. The $OIFS_HOME/AC-experiments/platform/<platform-name>-workflow.sh
templates provide examples that need to be adjusted for the local hardware and file paths. In this case the config.h
file only needs editing once and instead of the batch job wrapper script the workflow script is submitted.
- set the new environment variable: export OIFS_DATA_DIR=/data/openifs-ac/43r3
The following two tgz archives can be unpacked to any location (in this example the files will be retained in /data/openifs-ac):
- Unpack oifs-ac-43r3-initcond.tgz into a directory /data/openifs-ac/INITIAL-CONDITIONS. These files are initial data files for the first day of month for all 12 months of 2010.
- Unpack ICMCL-2010.tgz into a directory /data/openifs-ac/ICMCL. This directory contains monthly files with surface boundary conditions (e.g. SSTs, soil temps, albedos, LAI etc).
(Comment: Note that the ifsdata, wam and rdata directories are missing and need to be copied over from a standard OIFS_DATA_DIR installation. "wam" data directory is not standard for OpenIFS)
Building the model
On the ECMWF HPC Atos it is possible to compile the model using a login node. On the ECMWF Cray XC40 it is recommended to open an interactive session on a MOM node for compilation:
Code Block | ||||
---|---|---|---|---|
| ||||
% qsub -I -q dp -l EC_total_tasks=12 -l EC_job_tmpdir=10G -l EC_memory_per_task=3G |
If you have a configuration script that sets the environment for OpenIFS you should run this first.
In your command line you should type: export OIFS_CHEM=enable to overwrite the default setting in oifs.fcm (or alternatively you can edit the variable in file oifs.fcm)
Then compile the model using fcm:
No Format |
---|
cd $OIFS_HOME/make
fcm make -v -j 12 -f oifs.fcm |
Only proceed after the code has compiled without errors.
Run the acceptance tests
The model package offers various tests to check the basic functionality of the model and to verify that all paths to the data directories are correct and accessible.
These tests are run on a T21 full Gaussian grid and are designed to require a minimum of computational resources. Several different tests are being offered which activate different pathways in the model source code. It is highly recommended to carry out these brief tests prior to using the model.
Overview
To check the model is working normally carry out the various acceptance tests in $OIFS_HOME/t21test
and t21test_xios_t255ac
.
This requires editing the wrapper scripts job.sh
and setup-exp.sh
by setting the path to the appropriate master.exe and by changing RUNCMD to to something appropriate for the platform.
Info | ||
---|---|---|
| ||
When running on the ECMWF Cray XC40 the following changes need to be made:
|
t21test: This directory contains the original acceptance test which is provided with OpenIFS. It should work in the same way for OpenIFS/AC as described in the OpenIFS User Guide.
t21test_xios: This directory contains an acceptance test which makes use of the XIOS server – we do not use this here.
t21test_xios_t255ac: The three acceptance tests in this directory make use of the additional compositon (AC) code and it is possible to run this test (a) without AC as in t21test, (b) with additional AC included, and (c) with both additional AC and using the XIOS server.
Note |
---|
Note: If the XIOS server is to be used in these tests then a precompiled version of its libraries needs to be accessible and paths need to be set in oifs-config.me.sh. We do not describe here how XIOS is built, more information on XIOS can be found here. |
Step-by-step
Now we can carry out the tests:
t21test:
No Format |
---|
./job.sh -n 12 -t 2 -m ../make/gnu-opt-chem-fftw/oifs/bin/master.exe |
If the oifs_run script (or the oiac_run script) is used without the --nomove option it produces an error at the end of the acceptance test, stating the ICMGGepc8+* files cannot be found. This refers to the copying of model output to output1 and the postprocessing and is therefore not relevant to the model functionality.
t21test_xios_t255ac:
First run the standard t21test for experiment epc8. This test runs in directory $SCRATCH/43r3/epc8/
and hence the -x parameter needs to be given the absolute full path to the master.exe binary.
In setup-exp.sh
change the value for RUN_NUMBER
otherwise any pre-existing model output from this test will be overwritten.
No Format |
---|
./setup-exp.sh -n 16 -t 2 -x /home/damk/oifs/model/openifs-ac/make/gnu-opt-chem-fftw/oifs/bin/master.exe |
Now repeat the test with chemistry for experiment oiac. This test runs in directory $SCRATCH/43r3/oiac/
.
No Format |
---|
./setup-exp.sh -c -n 12 -t 2 -x /perm/rd/damk/oifs/openifs-ac/make/cce-opt/oifs/bin/master.exe |
This test runs with a 30 min time step and due to the computational requirements for the added chemistry it will take much longer to complete.
Note |
---|
Note: If the XIOS server is to be used in the t21test experiments (option |
Note: only tm5 chemistry seems to work, model fails with bascoetm5
Running OpenIFS/AC in Forecast or Climate Mode
(this section contains old stuff, move on to Steps to carry out an experiment)
The model is set up to run at Tl255L91 and it can be run in one of two modes:
Forecast mode: The model carries out a brief fc experiment with daily restart files. The intention is to restart the model every 1 or 2 days with new meteorological initial data obtained from ea or od, while the chemical fields are retained from the restart file in order to simulate the evolution of the atmospheric chemistry with time.
Climate mode: The model is run continuously for a longer period fc experiment while the dynamics is nudged every 6 hours to ei analyses (which were interpolated from 60 to 91 levels).
The default experiment ID is set as abcd and the model is set up such that it runs on $SCRATCH of the HPC.
$SCRATCH/43r3/experiments/abcd/chem_ic/
will contain the chemical initial and boundary conditions for the experiment.
$SCRATCH/43r3/experiments/abcd/runs/
will contain the experiment directory.
Steps to carry out an experiment
In $OIFS_HOME/AC-experiments/platform:
- create a machine config file: oifs-config.me.sh (we can re-use the one used in t21test_xios_t255ac)
- create a wrapper script to submit the batch job; the following templates exist:
- ecmwf-cca.job.tmpl – run oifs without XIOS (try this first!) this is virtually the same as $OIFS_HOME/AC-experiments/run-cca.job
- ecmwf-cca+xios.job.tmpl – run oifs with XIOS this is virtually the same as $OIFS_HOME/AC-experiments/run-cca+xios.job
- ecmwf-cca-workflow.sh – workflow for a run with a series of restarts
We begin with a batch job to run OpenIFS/AC without restarts and without XIOS. For this purpose we modify run-cca.job:
- in oifs-run.sh change platform to 'me' (or 'damk' in this case)
- in run-cca.job delete the accounting PBS header line:
#PBS -l EC_billing_account=nlchekli
Edit the following lines in file config.h:
- OIFS_DATA_DIR= (set to your oifs data dir)
- ICMCLDIR= (set to where ICMCL directory has previously been extracted to
- OIFS_INIDATA_DIR= path_to_.../INITIAL_CONDITIONS/T255L91-tm5-none
- MET_IC_DIR= path_to_...ICMCL/aabg/2010010100
Prepare initial conditions for chemistry:
- scripts require substring and grib_get commands: load modules sms and eccodes
- run script
prep-ic-icmcl-compo.sh 2010-01-01 2010-01-03
(if startdate and enddate are not given 2010-01-01 + 2 days fclength is used as default)
(this command takes a few minutes to complete) - during the above process an experiment directory is created:
$SCRATCH/43r3/experiments/{expid}/chem_ic/
with default expid=abcd (contains about 1.7 GB data)
Following these steps:
- submit job:
qsub ./run-cca.job
Script modifications
config.h
No Format |
---|
diff --git a/AC-experiments/config.h b/AC-experiments/config.h index 02a6f11..6fbf4fb 100644
--- a/AC-experiments/config.h
+++ b/AC-experiments/config.h
@@ -57,7 +57,7 @@ LAC4IC=true
# DATASETS
#==========================================================
# Base OIFS dataset
-OIFS_DATA_DIR="/scratch/ms/nl/nm6/ECE-DATA/OPENIFS/43r3"
+OIFS_DATA_DIR=/fwsm/lb/project/openifs/openifs-ac/43r3
# Nudging files location (must end with a "/") nudg_gg_file_path=/scratch/ms/nl/nm6/ECE-DATA/atm-nudging/oifs/
@@ -67,13 +67,13 @@ nudg_sh_file_path=/scratch/ms/nl/nm6/ECE-DATA/atm-nudging/oifs/
# ICMCL-YYYYMM0100, where YYYY and MM match those of your start date.
# If ICMCLDIR is empty or not defined, ICMCL files are created according to # section "ICMCL configuration" of config.h -ICMCLDIR=/scratch/ms/nl/nm6/ECE-DATA/OPENIFS/ICMCL
+ICMCLDIR=/fwsm/lb/project/openifs/openifs-ac/ICMCL
#### AC-specific: Dataset for IC and ICMCL-COMPO #### # # (1) Dir with already prepared IC and ICMCL-COMPO
# (Use option '-p' when calling "prep-ic-icmcl-compo.sh" script)
-OIFS_INIDATA_DIR="/scratch/ms/nl/nm6/ECE-DATA/OPENIFS/INITIAL-CONDITIONS/T255L91-tm5-none"
+OIFS_INIDATA_DIR="/fwsm/lb/project/openifs/openifs-ac/INITIAL-CONDITIONS/T255L91-tm5-none"
#
# (2) If files in OIFS_INIDATA_DIR are not used, IC and ICMCL-COMPO are
# created from archived experiments and distributed data.
@@ -83,7 +83,7 @@ OIFS_INIDATA_DIR="/scratch/ms/nl/nm6/ECE-DATA/OPENIFS/INITIAL-CONDITIONS/T255L91
#
# - DIR with standard IC. If empty, retrieved from archived experiments
# defined hereafter (require MARS access then)
-MET_IC_DIR=/scratch/ms/nl/nm6/ECE-DATA/OPENIFS/ICMCL/aabg/2010010100
+MET_IC_DIR=/fwsm/lb/project/openifs/openifs-ac/ICMCL/aabg/2010010100
#
# - FILE with AC tracers and initial fluxes (can be created with LAC4IC=true
# by OIFS/AC runs). Must be passed at the command line to the |
oifs-run.sh
No Format |
---|
diff --git a/AC-experiments/oifs-run.sh b/AC-experiments/oifs-run.sh index c65b243..64da99f 100755
--- a/AC-experiments/oifs-run.sh
+++ b/AC-experiments/oifs-run.sh
@@ -28,7 +28,7 @@ source scripts/lib_chem_setup.sh
## OIFS_NPROC : value set by the calling job
## OMP_NUM_THREADS : value set by the calling job
-platform='ecmwf-cca'
+platform='damk'
source platform/oifs-config.${platform}.sh
# ----------------------------------------------------------------------------- |
run-cca.job
...