This page describes the required steps to use OpenIFS/AC "v2ac-rc" on the ECMWF HPCF.
OpenIFS/AC compared to the standard OpenIFS model
OpenIFS/AC consists of the standard OpenIFS model and the additional composition software package. Both components together are referred to as OpenIFS/AC (with AC standing for "atmospheric composition").
The standard OpenIFS model is described in detail in the OpenIFS User Guide. The user guide introduces the model's components and supported model grids. It informs about additional required software packages and how to build the OpenIFS model. It further describes how to carry out an acceptance test after a successful model build in order to verify the model's functionality.
Users who are new to OpenIFS should first read the OpenIFS User Guide's sections 1 to 5 before proceeding with OpenIFS/AC, as the steps described therein are required for building OpenIFS/AC. The instructions in the user guide will not be repeated here.
Additional model sources
In the standard OpenIFS model the source code for atmospheric composition (atmospheric chemistry and aerosol code) has been replaced by dummy routines. The additional model sources for OpenIFS/AC replace these dummy routines and substitute the full composition source code. It is therefore necessary to rebuild the model executable after adding the additional model source code. This process is described further below.
Additional data packages
In order to run OpenIFS with atmospheric composition a substantial number of additional input data files are required. These provide initial conditions for the chemical and aerosol fields as well as additional boundary conditions for the model, mostly in the form of emission fluxes for a number of chemical tracers. These emissions are released mostly at the model surface, however they can also be associated with emission heights (e.g. for engine emissions from global air traffic). Further information is required for the physical removal of tracers through either wet scavenging or dry deposition, the latter of which is related to the surface vegetation type as used by the land surface scheme.
The additional data packages required for atmospheric composition modelling need to be prepared for the model at the desired grid resolution. While the standard OpenIFS model can be used with a wide range of model grids, the prepared chemical input data is only available for few selected grid resolutions and for specific time periods. If the model is to be used for other time periods or at different horizontal or vertical grid resolutions then the additional data packages need first to be generated for this purpose.
Note that the additional data packages required for OpenIFS/AC are only made available for specific time periods and grid resolutions.
Obtaining and compiling the code
Working with the ftp server
In order to convert the standard OpenIFS model to OpenIFS/AC the additional software package for composition needs to be installed into an existing OpenIFS installation.
If both standard (meteorological) model and the atmospheric composition model are to be used at the same time, it is recommended to create a new OIFS_HOME directory for OpenIFS/AC and to first unpack the standard OpenIFS model sources into this directory (for instance, if the standard OpenIFS model is installed into directory 'oifs43r3', the new destination for the model sources could be extracted into directory 'openifs-ac').
Once the standard OpenIFS model code is unpacked, unpack package <package-name> into $OIFS_HOME, overwriting any pre-existing files from the standard model installation.
TO DO: we need a separate code package for the OpenIFS/AC code that can be unpacked into the existing OpenIFS 43r3v2 source code distribution
Working with the git repository
OpenIFS/AC is maintained in Philippe Le Sager's ECMWF git repository: https://git.ecmwf.int/users/nm6
git clone ssh://git@git.ecmwf.int/~nm6/oifs43r3.git mv oifs43r3 openifs-ac cd openifs-ac git checkout v2ac-rc
Environment settings
In order to build OpenIFS/AC the standard environment settings for oifs43r3v2 can be used with the application of two changes:
- change OIFS_HOME to point to the installation of openifs-ac (and not to the standard oifs43r3 installation)
- change OIFS_DATA_DIR to a bespoke path as the additional data packages will be added to the climate data files; we describe below what date needs to be moved to this directory
- before compiling the model sources edit $OIFS_HOME/make/oifs.fcm to enable chemistry (which is by default disabled)
It is recommend for the first two changes to use a custom configuration script to set the environment variables.
Alternatively the settings can be enabled when running the prep scripts that come with OpenIFS/AC: For the t21 acceptance tests, the environment variables and paths are set within the oifs-config.<my_name>.sh which needs to be edited in the platform
subdirectory. This is described below.
Obtaining additional data packages
- Create a new directory which will contain (as a subdirectory) the new OIFS_DATA_DIR. We call it /data/openifs-ac.
Download the additional data tarballs that are required for chemistry experiments into /data/openifs-ac:
cd /data/openifs-ac module load ecfs ecp ec:/nm6/oifs/oifs-ac-43r3-chem.tgz . ecp ec:/nm6/oifs/oifs-ac-43r3-xtra.tgz . ecp ec:/nm6/oifs/oifs-ac-43r3-initcond.tgz . ecp ec:/nm6/oifs/ICMCL-2010.tgz .
- Unpack the following files: oifs-ac-43r3-chem.tgz and oifs-ac-43r3-xtra.tgz into /data/openifs. This will result in the following directory structure:
43r3/chemistry | AC specific |
43r3/xdata | AC specific |
43r3/rtables | Standard dataset for OIFS-43r3 |
43r3/climate.v015 | Standard dataset for OIFS-43r3 + AC specifics |
43r3/ifsdata | Standard dataset for OIFS-43r3 |
43r3/wam | Standard dataset for OIFS-43r3, needs to be added explicitly |
- set the new environment variable: export OIFS_DATA_DIR=/data/openifs-ac/43r3
The following two tgz archives can be unpacked to any location (in this example the files will be retained in /data/openifs-ac):
- Unpack oifs-ac-43r3-initcond.tgz into a directory /data/openifs-ac/INITIAL-CONDITIONS. These files are initial data files for the first day of month for all 12 months of 2010.
- Unpack ICMCL-2010.tgz into a directory /data/openifs-ac/ICMCL. This directory contains monthly files with surface boundary conditions (e.g. SSTs, soil temps, albedos, LAI etc).
(Comment: Note that the ifsdata, wam and rdata directories are missing and need to be copied over from a standard OIFS_DATA_DIR installation. "wam" data directory is not standard for OpenIFS)
Building the model
It is recommended to open an interactive session on a MOM node for compilation:
% qsub -I -q dp -l EC_total_tasks=12 -l EC_job_tmpdir=10G -l EC_memory_per_task=3G
If you have a configuration script that sets the environment run this first. Then edit oifs.fcm and enable chemistry as mentioned above. Then compile the model as usual using fcm:
cd $OIFS_HOME/make fcm make -v -j 6 -f oifs.fcm
The code should compile without errors when using CCE compilers.
Run the acceptance tests
The model package offers various tests to check the basic functionality of the model and to verify that all paths to the data directories are correct and accessible.
These tests are run on a T21 full Gaussian grid and are designed to require a minimum of computational resources. Several different tests are being offered which activate different pathways in the model source code. It is highly recommended to carry out these brief tests prior to using the model.
Step-by-step
To check the model is working normally carry out the various acceptance tests in $OIFS_HOME/t21test
and t21test_xios_t255ac
.
This requires editing the wrapper scripts job.sh
and setup-exp.sh
by setting the path to the appropriate master.exe and by changing RUNCMD to to something appropriate for the platform.
ECMWF Cray HPC
When running on the ECMWF Cray XC40 the following changes need to be made:
- In $OIFS_HOME/t21test/job.sh and t21test_xios_t255ac/setup-exp.sh include the line: export OIFS_RUNCMD="aprun -n $NPROC -d $NTHREADS -ss $MASTER". This variable has to be modified again when you want to use XIOS, I suggest to set it up outside the script, either at the command line during an interactive session or in a submit script (see t21test_xios_t255ac/platform/ecmwf-cca.job.tmpl and t21test_xios_t255ac/platform/ecmwf-cca.job+xios.tmpl).
See also here: Running with MPI on the Cray - aprun command - In $OIFS_HOME/t21test_xios_t255ac/platform/ copy file oifs-config.ecmwf-cca.sh to oifs-config.me.sh, open the latter file in an editor and change the variable settings and paths according to your environment. In $OIFS_HOME/t21test_xois_t255ac/setup-exp.sh edit the line: platform=me
t21test: This directory contains the original acceptance test which is provided with OpenIFS. It should work in the same way for OpenIFS/AC as described in the OpenIFS User Guide.
t21test_xios: This directory contains an acceptance test which makes use of the XIOS server – we do not use this here.
t21test_xios_t255ac: The three acceptance tests in this directory make use of the additional compositon (AC) code and it is possible to run this test (a) without AC as in t21test, (b) with additional AC included, and (c) with both additional AC and using the XIOS server.
Note: If the XIOS server is to be used in these tests then a precompiled version of its libraries needs to be accessible and paths need to be set in oifs-config.me.sh. We do not describe here how XIOS is built, more information on XIOS can be found here.
Now we can carry out the tests:
t21test:
./job.sh -n 12 -t 2 -m ../make/gnu-opt-chem-fftw/oifs/bin/master.exe
If the oifs_run script (or the oiac_run script) is used without the --nomove option it produces an error at the end of the acceptance test, stating the ICMGGepc8+* files cannot be found. This refers to the copying of model output to output1 and the postprocessing and is therefore not relevant to the model functionality.
t21test_xios_t255ac:
First run the standard t21test for experiment epc8. This test runs in directory $SCRATCH/43r3/epc8/
and hence the -x parameter needs to be given the absolute full path to the master.exe binary.
In setup-exp.sh
change the value for RUN_NUMBER
otherwise any pre-existing model output from this test will be overwritten.
./setup-exp.sh -n 16 -t 2 -x /home/damk/oifs/model/openifs-ac/make/gnu-opt-chem-fftw/oifs/bin/master.exe
Now repeat the test with chemistry for experiment oiac. This test runs in directory $SCRATCH/43r3/oiac/
.
./setup-exp.sh -c -n 12 -t 2 -x /perm/rd/damk/oifs/openifs-ac/make/cce-opt/oifs/bin/master.exe
This test runs with a 30 min time step and due to the computational requirements for the added chemistry it will take much longer to complete.
Note: If the XIOS server is to be used in the t21test experiments (option -i
) then the variables OIFS_XIOS
and OIFS_XIOS_DIR
need to be set appropriately in the configuration file in the platform
directory. Also, the binary executable master.exe
needs to have been built with XIOS support enabled.
Note: only tm5 chemistry seems to work, model fails with bascoetm5
Running OpenIFS/AC in Forecast or Climate Mode
(this section contains old stuff, move on to Steps to carry out an experiment)
The model is set up to run at Tl255L91 and it can be run in one of two modes:
Forecast mode: The model carries out a brief fc experiment with daily restart files. The intention is to restart the model every 1 or 2 days with new meteorological initial data obtained from ea or od, while the chemical fields are retained from the restart file in order to simulate the evolution of the atmospheric chemistry with time.
Climate mode: The model is run continuously for a longer period fc experiment while the dynamics is nudged every 6 hours to ei analyses (which were interpolated from 60 to 91 levels).
The default experiment ID is set as abcd and the model is set up such that it runs on $SCRATCH of the HPC.
$SCRATCH/43r3/experiments/abcd/chem_ic/
will contain the chemical initial and boundary conditions for the experiment.
$SCRATCH/43r3/experiments/abcd/runs/
will contain the experiment directory.
Steps to carry out an experiment
In $OIFS_HOME/AC-experiments/platform:
- create a machine config file: oifs-config.me.sh (we can re-use the one used in t21test_xios_t255ac)
- create a wrapper script to submit the batch job; the following templates exist:
- ecmwf-cca.job.tmpl – run oifs without XIOS (try this first!) this is virtually the same as $OIFS_HOME/AC-experiments/run-cca.job
- ecmwf-cca+xios.job.tmpl – run oifs with XIOS this is virtually the same as $OIFS_HOME/AC-experiments/run-cca+xios.job
- ecmwf-cca-workflow.sh – workflow for a run with a series of restarts
We begin with a batch job to run OpenIFS/AC without restarts and without XIOS. For this purpose we modify run-cca.job:
- in oifs-run.sh change platform to 'me' (or 'damk' in this case)
- in run-cca.job delete the accounting PBS header line:
#PBS -l EC_billing_account=nlchekli
Edit the following lines in file config.h:
- OIFS_DATA_DIR= (set to your oifs data dir)
- ICMCLDIR= (set to where ICMCL directory has previously been extracted to
- OIFS_INIDATA_DIR= path_to_.../INITIAL_CONDITIONS/T255L91-tm5-none
- MET_IC_DIR= path_to_...ICMCL/aabg/2010010100
Prepare initial conditions for chemistry:
- scripts require substring and grib_get commands: load modules sms and eccodes
- run script
prep-ic-icmcl-compo.sh 2010-01-01 2010-01-03
(if startdate and enddate are not given 2010-01-01 + 2 days fclength is used as default)
(this command takes a few minutes to complete) - during the above process an experiment directory is created:
$SCRATCH/43r3/experiments/{expid}/chem_ic/
with default expid=abcd (contains about 1.7 GB data)
Following these steps:
- submit job:
qsub ./run-cca.job
Script modifications
config.h
diff --git a/AC-experiments/config.h b/AC-experiments/config.h index 02a6f11..6fbf4fb 100644 --- a/AC-experiments/config.h +++ b/AC-experiments/config.h @@ -57,7 +57,7 @@ LAC4IC=true # DATASETS #========================================================== # Base OIFS dataset -OIFS_DATA_DIR="/scratch/ms/nl/nm6/ECE-DATA/OPENIFS/43r3" +OIFS_DATA_DIR=/fwsm/lb/project/openifs/openifs-ac/43r3 # Nudging files location (must end with a "/") nudg_gg_file_path=/scratch/ms/nl/nm6/ECE-DATA/atm-nudging/oifs/ @@ -67,13 +67,13 @@ nudg_sh_file_path=/scratch/ms/nl/nm6/ECE-DATA/atm-nudging/oifs/ # ICMCL-YYYYMM0100, where YYYY and MM match those of your start date. # If ICMCLDIR is empty or not defined, ICMCL files are created according to # section "ICMCL configuration" of config.h -ICMCLDIR=/scratch/ms/nl/nm6/ECE-DATA/OPENIFS/ICMCL +ICMCLDIR=/fwsm/lb/project/openifs/openifs-ac/ICMCL #### AC-specific: Dataset for IC and ICMCL-COMPO #### # # (1) Dir with already prepared IC and ICMCL-COMPO # (Use option '-p' when calling "prep-ic-icmcl-compo.sh" script) -OIFS_INIDATA_DIR="/scratch/ms/nl/nm6/ECE-DATA/OPENIFS/INITIAL-CONDITIONS/T255L91-tm5-none" +OIFS_INIDATA_DIR="/fwsm/lb/project/openifs/openifs-ac/INITIAL-CONDITIONS/T255L91-tm5-none" # # (2) If files in OIFS_INIDATA_DIR are not used, IC and ICMCL-COMPO are # created from archived experiments and distributed data. @@ -83,7 +83,7 @@ OIFS_INIDATA_DIR="/scratch/ms/nl/nm6/ECE-DATA/OPENIFS/INITIAL-CONDITIONS/T255L91 # # - DIR with standard IC. If empty, retrieved from archived experiments # defined hereafter (require MARS access then) -MET_IC_DIR=/scratch/ms/nl/nm6/ECE-DATA/OPENIFS/ICMCL/aabg/2010010100 +MET_IC_DIR=/fwsm/lb/project/openifs/openifs-ac/ICMCL/aabg/2010010100 # # - FILE with AC tracers and initial fluxes (can be created with LAC4IC=true # by OIFS/AC runs). Must be passed at the command line to the
oifs-run.sh
diff --git a/AC-experiments/oifs-run.sh b/AC-experiments/oifs-run.sh index c65b243..64da99f 100755 --- a/AC-experiments/oifs-run.sh +++ b/AC-experiments/oifs-run.sh @@ -28,7 +28,7 @@ source scripts/lib_chem_setup.sh ## OIFS_NPROC : value set by the calling job ## OMP_NUM_THREADS : value set by the calling job -platform='ecmwf-cca' +platform='damk' source platform/oifs-config.${platform}.sh # -----------------------------------------------------------------------------
run-cca.job
diff --git a/AC-experiments/run-cca.job b/AC-experiments/run-cca.job index f4d09a3..db5c21f 100644 --- a/AC-experiments/run-cca.job +++ b/AC-experiments/run-cca.job @@ -1,5 +1,4 @@ #PBS -q np -#PBS -l EC_billing_account=nlchekli #PBS -j oe #PBS -S /bin/bash #PBS -l EC_total_tasks=288