Note |
---|
Note that this documentation applies to the OpenIFS/AC code extension for OpenIFS 43r3. This does not apply to users of OpenIFS 48r1 or later model releases. |
...
Table of Contents |
---|
...
This page describes how OpenIFS/AC v1 (derived from OpenIFS 43r3v2) can be obtained, built and used for forecast experiments. The documentation shows examples of how to use the model on the ECMWF Atos Sequana XH2000 HPCF (hpc2020). Adjustments will need to be made for local installations.
Info |
---|
Enquiries about OpenIFS/AC or technical questions should in the first instance be sent to openifs-support@ecmwf.int |
OpenIFS/AC compared to the standard OpenIFS model
...
The standard OpenIFS model is described in detail in the OpenIFS 43r3 User Guide. The user guide introduces the model's components and supported model grids. It informs about additional required software packages and how to build the OpenIFS model. It further describes how to carry out an acceptance test after a successful model build in order to verify the model's functionality.
Info |
---|
Users who are new to OpenIFS should first read the OpenIFS 43r3 User Guide's sections 1 to 5 before proceeding with OpenIFS/AC, as the steps described therein are required for building OpenIFS/AC. The instructions in the user guide will not be repeated here. |
...
No Format |
---|
ftp ftp.ecmwf.int (login with username and password for OpenIFS users) cd src/openifs-ac get openifs-ac.43r3.v1.tgzgz bye |
Installing the OpenIFS/AC sources
...
- Create a new home for the OpenIFS/AC installation. Here we call this location /home/openifs-ac.
- Change into your new directory and unpack therein the original OpenIFS model sources: tar xvzf oifs43r3v2.tgz
- In a second step, in the same folder unpack gunzip the OpenIFS/AC sources : tar xvzf and patch the original OpenIFS:
- gunzip openifs-ac.43r3.v1.
- gz
- patch -p1 <openifs-ac.43r3.v1
Info |
---|
Note that some files of the original model will be replacedmodified. It is therefore essential to install the original OpenIFS 43r3v2 sources first before installing the OpenIFS/AC sources. |
...
- Change into your $OIFS_HOME directory where OpenIFS is installed: cd $OIFS_HOME
- In this directory extract the downloaded source package: tar xvzf
- gunzip openifs-ac.43r3.v1
- .gz
- patch -p1 <openifs-ac.43r3.v1
Info |
---|
Note that this unpacking will not only add new files to your existing $OIFS_HOME but it will also overwrite some of the existing files with updated versions required for OpenIFS/AC. |
...
- Unpack oifs-ac-43r3-initcond.tgz into /data/openifs-ac which results in a directory /data/openifs-ac/INITIAL-CONDITIONS.
The files therein are initial data files for the first day of month for all 12 months of 2010. - Unpack ICMCL-2010.tgz into /data/openifs-ac which results in a directory /data/openifs-ac/ICMCL.
This directory contains monthly files with surface boundary conditions (e.g. SSTs, soil temps, albedos, LAI etc). - In the tgz archives above, the data for the year 2013, provided on a Tl255L60 grid, is required for the T21 acceptance tests. The data for year 2010 is provided on a Tl255L91 grid and can be used for the example forecasts.
- From your conventional OpenIFS installation's
OIFS_DATA_DIR
copy the directories ifsdata and rtables to /data/openifs-ac/43r3 as they are also needed here. If this is your first model installation, these data directories are also available from the ftp server. - Further, create a directory /data/openifs-ac/43r3/wam which needs to contain wave model initial data for 2010-01-01.
Building the model
Info |
---|
Note: In the reference forecast experiments discussed below the initial conditions files for the wave model (ocean surface waves) are added to the experiment separately from a central location in the |
...
Building the model
This section describes how to build the binary model This section describes how to build the binary model executable for OpenIFS/AC. Most problems related to building the model executable are due to incorrect environment settings. Therefore we recommend that particular care should be given to the environment settings to ensure that they match the local hardware platform.
...
- To do this we run the standard t21test for experiment epc8 (T21L19) with the
setup-exp.sh
new script. - The name for the hardware platform needs to be set (l.89, e.g.
platform=ecmwf-hpc2020
) - The test experiment runs in the
RUNDIR
directory specified insetup-exp.sh
(l.94, test experiment runs in theRUNDIR
directory specified insetup-exp.sh
( the default is$SCRATCH/43r3/epc8/
). - We also need to use the command line option -x and give as argument the absolute full path to the
master.exe
binary executable. - In
setup-exp.sh
you should change the value forRUN_NUMBER
otherwise any pre-existing model output from this test will be overwritten.
...
Test result: As before an error calculation file res_021_0072
is created in the run directory and the model output is found in the subfolder output1
(this name depends on the RUN_NUMBER variable).
With chemistry: Now repeat the test with enabled model chemistry. This is a short 1-day test run that uses the newly added source code of OpenIFS/AC. It has a very limited set of options and should not be used for forecast experiments. Note that you cannot increase the experiment duration beyond one day because the boundary conditions are only supplied from 2013-07-01 to 2013-07-03.
Info |
---|
Running on the ECMWF HPCF: |
With chemistry: Now repeat the test with enabled model chemistry. This is a short 1-day test run that uses the newly added source code of OpenIFS/AC. It has a very limited set of options and should not be used for forecast experiments. Note that you cannot increase the experiment duration beyond one day because the boundary conditions are only supplied from 2013-07-01 to 2013-07-03.
This experiment is named oiac
and uses a Tl255L60 This experiment is named oiac
and uses a Tl255L60 grid and runs for 24 hours. The default run directory is $SCRATCH/43r3/oiac/
.
...
Test result: This experiment calculates budgets of atmospheric mass and ozone which are stored , along with overall production and loss tendencies, as well as more detailed reaction budget terms relevant for the ozone production and loss tendency. These are printed in the NODE.001_01
file and also in afterwards extracted to the files $RUNDIR/output{$RUN_NUMBER}/massdia_chem_oiac.txt
and ozonbud_chem_dia.txt
.
...
No Format |
---|
EXPID=oiac; RUNDIR=${SCRATCH}/${OIFS_CYCLE}/${EXPID} ; RUN_NUMBER=3 #checkoifs fref1=${OIFS_HOME}/t21test_xios_ac/ctrl/massdia_chem__oiac_2013070100.txt fref2=${OIFS_HOME}/t21test_xios_ac/ctrl/ozonbud_chem__oiac_2013070100.txt fnew1=${RUNDIR}/output${RUN_NUMBER}/massdia_chem_oiac.txt fnew2=${RUNDIR}/output${RUN_NUMBER}/ozonbud_chem_oiac.txt diff $fref1 $fnew1 diff $fref2 $fnew2 |
Differences As a rule of thumb, differences between the reference budgets budget numbers and those from the t21test should be only small (below 5%), although this depends on the exact variable that is assessed.
...
Single forecast experiments
This section describes the required steps to carry out a single forecast experiment (no restarts) with chemistry. The essential three steps can be summarised as:
- Edit paths and variables in
$OIFS_HOME/AC-experiments/config.h
. This controls model switches and the chemistry scheme. - Prepare initial conditions using
$OIFS_HOME/AC-epxeriments/prep-ic-icmcl-compo.sh
. - Customise a job wrapper script for the relevant hardware platform (a
*.job
file) in order to submit theoifs-run.sh
script to the batch scheduler.
Overview of the AC-experiments directory
...
No Format |
---|
AC-experiments/ ├── config.h # MAIN experiment config file ├── ctrl/ │ ├── context_ifs.xml # xios resource │ ├── ifs_xml │ / │ ├── axis_def_ifs.xml │ │ ├── domain_def_ifs.xml │ │ ├── field_def_ifs.xml │ │ ├── file_def_ifs.xml │ │ └── grid_def_ifs.xml # xios resource │ ├── iodef.xml # xios resource │ ├── namelistfc │ ├── namelist.ifs-epc8.sh │ ├── namelist.ifs-m7-debug-output.sh# namelist for t21test without chemistry │ ├── namelist.ifs-oiac.sh │ ├──# namelist.ifs.sh │ ├── Table for t21test with chemistry │ │ ├── bins_aerver1.csv │ │ ├── bins_hamm7ver1.csv │ │ ├── bins_tm5m7ver0.csv │ │ ├── bins_tm5m7ver1.csv │ │ ├── table_bascoetm5ver2d.txt │ │ ├── table_bascoetm5ver2f.txt │ │ ├── table_hamm7ver1.txt │ │ ├── table_RnPbver1.txt │ │ ├── table_tm5ver15.txt │ │ └── table_tm5verb27y.txtnamelist.ifs.sh # namelist template for forecast experiments │ ├── Table/ # dir with AC tracers │ └── wam_namelist ├── ecmwf-atoshpc2020.job ├── oifs-run.sh ├── platform │ ├── climate-prep.puhti.job │ ├── climate-run.puhti.job │ ├── csc-puhti.job.tmpl │ ├── ecmwf-cca.job.tmpl │ ├── ecmwf-cca-workflow.sh │ ├── ecmwf-cca+xios.job.tmpl │ ├── ecmwf-hpc2020.job.tmpl │ ├── knmi-rhino.job.tmpl │ ├── knmi-rhino-workflow.sh │ ├── oifs-config.csc-puhti.sh │ ├── oifs-config.ecmwf-atos.sh │ ├── oifs-config.ecmwf-cca.sh │ ├── oifs-config.ecmwf-hpc2020.sh │ ├── oifs-config.knmi-rhino.sh │ ├── prep-ic-icmcl-compo-puhti.job.tmpl │ └── README_PLATFORM ├── prep_emis │ ├── check_emis.py │ ├── log │ │ ├── CMIP6_MACCity_gfas_rean_v2_burden.txt │ │ ├── CMIP6_MACCity_gfas_rean_v2.pdf │ │ └── CMIP6_MACCity_gfas_rean_v2.txt │ ├── prepare_cb05aer_v1.0 │ ├── prep_cmip6_v0.1 │ └── Table │ └── table_tm5ver15_aer.txt # job wrapper script for oifs-run.sh ├── oifs-run.sh # MAIN run script ├── platform/ # dir with platform-specific templates ├── prep_emis/ # dir with emission utilities ├── prep-ic-icmcl-compo.sh # MAIN script to prepare initial/boundary conditions └── scripts/ # dir with low-level scripts ├── add_nrt_fire_chem ├── gaussgr ├── get_alb_lai_intLimits ├── get_tablecol ├── grib_def.h ├── lib_chem_setup.sh ├── lib_general.sh ├──# lib_icmcl_clima.sh of utilities ├── lib_icmcl_compoclima.sh ├── # lib of functs to create ICMCL boundary cond. files ├── lib_icmcl_compo.sh # lib of functs to create ICMCL-COMPO files ├── lib_initcond_offline.sh └── lib_initcond.sh # lib of functs to create initial conditions files |
The experiment controls are in $OIFS_HOME/AC-experiments/.
There the following scripts essential files can be found:
- prep-ic-icmcl-compo.sh - a script to generate initial conditions for chemical fields for the selected run time. Begin and end date need to be given as an argument.
- config.h - settings for model installation, paths, and experiment configuration.
- oifs-run.sh - this script controls the workflow for the experiment: reads experiment config, sets up run directories (in $SCRATCH), loads platform configuration, edits namelists, links to initial conditions and executes OIFS_RUNCMD
- ecmwf-atoshpc2020.job wrapper script for oifs-run.sh, required settings for the batch scheduler
...
- it is necessary to create a platform-specific machine config file: oifs-config.ecmwf-atoshpc2020.sh
The above ecmwf-atoshpcs2020.job wrapper script to submit the batch job is based on templates in the platform directory; a number of templates exist and for the ECMWF HPCF the following templates existare relevant:
- ecmwf-hpc2020.job.tmpl – run oifs without XIOS
- oifs-config.ecmwf-hpc2020.sh – template for the machine config file
...
No Format |
---|
PLATFORM='ecmwf-atoshpc2020' # this needs to match the name of the platform configuration file EXP=abcd # you should select here an experiment ID (4-letter alphanumeric word, must not start with '0') RUNDIR=${SCRATCH}/43r3/experiments/${EXP}/runs # this is the experiment directory AC_IC_DIR=${SCRATCH}/43r3/experiments/${EXP}/chem_ic # this is the stageingstaging directory for initial and boundary conditions data files # Simulation start and end date. If setting time (not mandatory), **put it before** the date run_start_date="00:00 2010-01-01" run_end_date="${run_start_date} + 2 days" rst_freq='6 day' # Run for 6 days without restarts. Our experiment will finish after 2 days, so this will be ignored. USE_RESTART=false # Do not write restart files. |
...
No Format |
---|
OIFS_INIDATA_DIR=${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-tm5-aer # Initialinitial and boundary conditions for the chemistry scheme used ICMCLDIR=${OIFS_DATA_DIR}/ICMCL # physical/meteorological boundary conditions MET_IC_DIR=${OIFS_DATA_DIR}/ICMCL/aabg/2010010100 # path for meteorological initial data in case it needs to be generated AC_IC_ONCE=${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-tm5-aer/ICMUAoiac.ac_rst # initial conditions for chemical tracers |
...
Using TM5 chemistry without aerosol
Example: We will run Run a 2-day forecast starting from 2010-01-01 with TM5 chemistry without aerosol.
...
Code Block |
---|
PLATFORM='ecmwf-atoshpc2020' RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs' # where your experiment will run AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic' # where your chemical initial conditions will be created run_end_date="2010-01-03" rst_freq='6 day' USE_RESTART=false LCHEM=true CHEM_SCHEME="tm5" CHEM_VER="ver15" OIFS_INIDATA_DIR=$path_to_directory_'${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-tm5-none AC_IC_ONCE=${OIFS_INIDATA_DIR}/ICMUAoiac.ac_rst' MET_IC_DIR='${OIFS_ICMCL_DIR}/aabg/2010010100' LAERO=false ' LAERO=false LAERCHEM=false LCHEM_AEROI=false LINJ_CHEM=false LINJ_AER=false LAEROSFC=false LWAM=true LRLXG=false LAC4IC=true |
Using TM5 chemistry with AER aerosol
Example: We will run Run a 2-day forecast starting from 2010-01-01 with TM5 chemistry and the AER bulk aerosol scheme.
...
Code Block |
---|
PLATFORM='ecmwf-atoshpc2020' RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs' # where your experiment will run AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic' # where your chemical initial conditions will be created run_end_date="2010-01-03" rst_freq='6 day' USE_RESTART=false LCHEM=true CHEM_SCHEME="tm5" CHEM_VER="ver15" OIFS_INIDATA_DIR=$path_to_directory_'${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-tm5-aer AC_IC_ONCE=${OIFS_INIDATA_DIR}/ICMUAoiac.ac_rst' MET_IC_DIR='${OIFS_ICMCL_DIR}/aabg/2010010100' LAERO=true LAERCHEM=true LCHEM_AEROI=true LINJ_CHEM=true LINJ_AER=true ' LAERO=true LAERCHEM=true LCHEM_AEROI=true LINJ_CHEM=true LINJ_AER=true LAEROSFC=true LWAM=true LRLXG=false LAC4IC=true |
Using BASCOE-TM5 chemistry without aerosol
Example: We will run Run a-2 day forecast starting from 2010-01-01 with combined BASCOE and TM5 chemistry without aerosol.
...
Code Block |
---|
PLATFORM='ecmwf-atoshpc2020' RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs' # where your experiment will run AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic' # where your chemical initial conditions will be created run_end_date="2010-01-03" rst_freq='6 day' USE_RESTART=false LCHEM=true CHEM_SCHEME="bascoetm5" CHEM_VER="ver2f" OIFS_INIDATA_DIR=$path_to_directory_'${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-bascoetm5-none AC_IC_ONCE=${OIFS_INIDATA_DIR}/ICMUAoiac.ac_rst' MET_IC_DIR='${OIFS_ICMCL_DIR}/aabg/2010010100' LAERO=false LAERCHEM=false LCHEM_AEROI=false ' LAERO=false LAERCHEM=false LCHEM_AEROI=false LINJ_CHEM=false LINJ_AER=false LAEROSFC=false LWAM=true LRLXG=false LAC4IC=true |
Using BASCOE-TM5 chemistry with AER aerosol
Example: We will run Run a-2 day forecast starting from 2010-01-01 with combined BASCOE and TM5 chemistry without aerosol.
...
Code Block |
---|
PLATFORM='ecmwf-atoshpc2020' RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs' # where your experiment will run AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic' # where your chemical initial conditions will be created run_end_date="2010-01-03" rst_freq='6 day' USE_RESTART=false LCHEM=true CHEM_SCHEME="bascoetm5" CHEM_VER="ver2f" OIFS_INIDATA_DIR=$path_to_directory_'${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-bascoetm5-none AC_IC_ONCE=${OIFS_INIDATA_DIR}/ICMUAoiac.ac_rst' MET_IC_DIR='${OIFS_ICMCL_DIR}/aabg/2010010100' LAERO=true LAERCHEM=true LCHEM_AEROI=true LINJ_CHEM=true LINJ_' LAERO=true LAERCHEM=true LCHEM_AEROI=true LINJ_CHEM=true LINJ_AER=true LAEROSFC=true LWAM=true LRLXG=false LAC4IC=true |
Generating initial data
Initial conditions for the standard OpenIFS model can be generated using the OpenIFS Data Hub or alternatively they can be requested from the ECMWF OpenIFS support by email openifs-support@ecmwf.int. This provides initial experiment data and surface boundary conditions for the physical/meteorological NWP forecast model.
...
The prep-ic-icmcl-compo.sh script (henceforth: prep script) carries out 4 tasks:
- Create Creates a destination directory $AC_IC_DIR which serves as a staging area for all initial and boundary conditions required for the forecast experiment. Symbolic links will point later from inside the experiment run directory to these files.
- Copy Copies the physical/meteorological initial conditions files that must be specific to the experiment start date from $OIFS_INIDATA_DIR to the new $AC_IC_DIR. If these files cannot be found then they are createdan attempt is made to create them from files found in the $MET_IC_DIR directory.
- Copy Copies the physical boundary conditions file ICMCL{exp}INIT (SSTs, albedos, etc) from $OIFS_INIDATA_DIR to $AC_IC_DIR and if necessary truncate to length of experiment duration.
- Copy Copies the chemistry initial and boundary conditions file (emission fluxes, deposition velocities) ICMCL-INIT-COMPO from $OIFS_INIDATA_DIR to $AC_IC_DIR. If this file cannot be found it will be created.
The syntax to run the prep script is:
No Format | nopanel | true
---|
./prep-ic-icmcl-compo.sh <start_date> <end_date> |
with start_date and end_date having the format YYYY-MM-DD.
In the examples above for a 2-day forecast, the syntax should be: be ./prep-ic-icmcl-compo.sh 2010-01-01 2010-01-03
to run generate the data for 48 hours from 1st January 2010.
Running the prep script will take a few minutes and will result in new data which that will be written to folder $AC_IC_DIR
, by default this location is $SCRATCH/43r3/experiments/{exp}/chem_ic
.
This folder serves as a staging area for the initial experiment and boundary conditions data and if the target files in $AC_IC_DIR
already exist in this destination then they they will not be overwritten.
Running a forecast experiment on the ECMWF HPCF
The actual forecast experiment will be carried out inside the run directory. By default this is set to $SCRATCH/43r3/experiments/${exp}/rundir/
.
Multiple forecasts with restarts
Info |
---|
This option is at an experimental development stage and it is therefore currently not supported for general users. Its implementation depends on scripting that is dependant on the hardware platform. Suggestions for script templates that are used on the ECMWF HPCF can be provided on request. |
Further information
----- THE REMAINING TEXT IS TO BE DELETED PRIOR TO PUBLICATION -----
EADME
Creating initial conditions at the end of an experiment run
When setting LAC4IC=true
an AC-for-IC grib file is created at the end of a run. It is a snapshot of all chemical/atmospheric composition tracers. This file can be used by the pre-processing script (prep script) to create new initial conditions.
With the AC-for-IC file, initial conditions for a new experiment can be generated which will provide a continuation of the chemical evolution of the tracer fields from the previous experiment. The meteorological fields to complement the initial conditions can originate from an archived experiment or can be generated with the OpenIFS Data Hub. This workflow is desirable when aiming to reinitialise the model integration with an updated weather forecast or analysis data (which does not provide AC tracers).
In order to restart with atmospheric composition tracers from the AC-for-IC grib file select the following settings in config.h
:
- Either leave the variable
OIFS_INIDATA_DIR
empty, delete it or comment it out. - Set the variable
AC_IC_ONCE
to point to the generated AC-for-IC grib file from the previous experiment. This file is typically found in this location in the previous experiment:${RUNDIR}/output/${RUN_NUMBER}/ICMUA${EXP}.ac_rst
- Set the variable
MET_IC_DIR
to the relevant path for the new experiment's initial conditions or modify the "variables to retrieve METEO initial conditions from an archived experiment" section as appropriate.
Running the forecast experiment
Now the actual forecast experiment can be submitted to the batch scheduler using the platform-specific job wrapper script. Examples for this are found in $OIFS_HOME/AC-experiments/platform
. An example template for the ECMWF HPCF Atos Sequana XH2000 is the file ecmwf-hpc2020.job.tmpl
.
The actual forecast experiment will be carried out inside the run directory which is defined in config.h
with variable RUNDIR
. The default on the ECMWF HPCF is $SCRATCH/43r3/experiments/${exp}/runs/
.
Due to the much increased computational requirements for simulating atmospheric composition, forecast experiments are often broken into individual parts (sometimes referred to as "legs"). For each leg a restart of the forecast experiment needs to be carried out.
Model Output
The oifs-run.sh
script for OpenIFS/AC has been designed to be able to cope with restarts or "legs", and the model output from each experiment leg is stored in individual subfolders named by leg number. In the case of a single forecast experiment without restarts the output will be written to $RUNDIR/output/001
.
Log files
Log file and experiment-related configuration data (namelist, NODE file, ifs.stat), as well as text files with diagnostic model outuput (e.g. mass and ozone budgets) are copied to $RUNDIR/log/001
.
Restart Data
Model data relevant for restarts (not discussed here on this page) will be stored in $RUNDIR/restart/${next_leg_number}.
XIOS output
Files relevant to XIOS server output are stored in $RUNDIR/ifs_xml (not discussed here on this page).
...
Multiple forecasts with restarts
Info |
---|
Note: This option is at an experimental development stage and it is therefore currently not supported for general users. Its implementation depends on scripting that is dependant on the hardware platform. Suggestions for script templates that are used on the ECMWF HPCF can be provided on request. |
There are several reasons why a user may wish to restart a forecast experiment. The increase in computational costs for simulating atmospheric composition may require to break a longer forecast experiment into individual sections ("legs"). Also, a restart will allow to initialise the meteorology in the model with data taken from an analysis of the current modelled time.
Due to the time scales associated with the different chemical processes and the lifetime of chemical tracers with respect to their loss processes, it is often convenient to retain the chemical tracer fields in the model simulation and update only the driving meteorology at the time of the restart. In some applications, the model could even be initialised every single day with an up-to-date meteorological analysis field while the chemical fields are left to evolve. The previous section described how chemical initial conditions can be created at the end of an experiment run, thus retaining the chemical tracer information from the previous experiment leg.
We will here provide some examples of the principle steps required for restarting experiments.
Example 1: Longer experiment with manual stop and restart
The experiment parameters need to be defined by editing config.h
. The following settings are used for a 2-month experiment with a stop-and-restart after the first month. This will result in two experiment legs of 1 month duration each.
No Format |
---|
run_start_date="2010-01-01"
run_end_date="${run_start_date} + 2 months"
rst_freq="1 month"
USE_RESTART=true |
The initial conditions for each leg need to be either created or gathered from a prepared repository.
No Format |
---|
prep-ic-icmcl-compo.sh 2010-01-01 2010-02-01
prep-ic-icmcl-compo.sh 2010-02-01 2010-03-01 |
These processes can be run simultaneously. The meteorological fields and initial conditions can be requested from the OpenIFS Data Hub. The wrapper script needs to be submitted for the first experiment leg. Once the experiment has completed, config.h needs to be edited to use the initial experiment data from the date and time when the first leg has stopped. Then the wrapper script for the second month should be submitted.
Example 2: Using a workflow script
The 4 jobs in the previous example, can be all triggered automatically and in a timely fashion with a single script. The $OIFS_HOME/AC-experiments/platform/<platform-name>-workflow.sh
templates provide examples that need to be adjusted for the local hardware and file paths. In this case the config.h
file only needs editing once and instead of the batch job wrapper script the workflow script is submitted. ━━━━━━━━━━━━━━━━━━━━━━━━━
ATMOSPHERIC COMPOSITION
━━━━━━━━━━━━━━━━━━━━━━━━━
Table of Contents
─────────────────
1. Test data
.. 1. Datasets for OIFS/AC
.. 2. Overview table of provided Initial Conditions
2. t21test_xios_t255ac
.. 1. Presentation
.. 2. How to run
.. 3. Tested configurations
.. 4. Output and diagnostics
3. AC-experiments
.. 1. Overview
.. 2. Main scripts
..... 1. Pre-processing
..... 2. OIFS run
.. 3. Configuration
.. 4. Examples
..... 1. Example 1 - A simple run
..... 2. Example 2 - Longer experiment with stop-and-restart
..... 3. Example 3 - Using a workflow script
..... 4. About performance - Long pre-processing
.. 5. Creating ICs at the end of run
..... 1. Example - A series of daily runs
4. Output
5. Using the workflow without restart
━━━━━━━━━━━━━━━━━━━━━━━━━
There are two of directories for the Atmospheric Composition work. The
/t21test_xios_t255ac/ is just a test run with almost no options that you
can set. In /AC-experiments/ you can configure your experiment in
greater detail. In all cases, you need to install some additional data
for the AC work.
1 Test data
═══════════
1.1 Datasets for OIFS/AC
────────────────────────
• You can fetch the main data dedicated to AC configurations from ECFS:
┌────
│ ec:/nm6/oifs/oifs-ac-43r3-chem.tgz
│ ec:/nm6/oifs/oifs-ac-43r3-xtra.tgz
└────
This will unpack to three directories: 43r3/chemistry, 43r3/climate.v015,
and '43r3/xdata'. The second one adds input to the standard
'ifsdata_cy43r3_climate.v015_255l.tgz'
• Unpack them in the same directory where you unpack the default
OpenIFS data, which should then contain:
┌────
│ 43r3/chemistry # AC specific
│ 43r3/xdata # AC specific
│ 43r3/rtables # Standard dataset for OIFS-43r3
│ 43r3/climate.v015 # Standard dataset for OIFS-43r3 + AC specifics
│ 43r3/ifsdata # Standard dataset for OIFS-43r3
│ 43r3/wam # Standard dataset for OIFS-43r3
└────
• A set of initial conditions is available:
┌────
│ ec:/nm6/oifs/oifs-ac-43r3-initcond.tgz
└────
Unpack into a directory: INITIAL-CONDITIONS
• A set of ICMCL climatology for the years 2010 on the TL255L90 grid (kindly
prepared by ECMWF) and few days of 2013 on TL255L60 is also needed:
┌────
│ ec:/nm6/oifs/ICMCL-2010.tgz
└────
Unpack into a directory: ICMCL
1.2 Overview table of provided Initial Conditions
─────────────────────────────────────────────────
The tested and supported configurations are:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
RESOL LEVELS LCHEM CHEM_SCHEME LAERO AERO_SCHEME START_DATE
───────────────────────────────────────────────────────────────────────────────────────────
T255L60-bascoetm5-aer 255 60 true bascoetm5 true aer 2013-07-01
T255L60-tm5-aer 255 60 true tm5 true aer 2013-07-01
───────────────────────────────────────────────────────────────────────────────────────────
T255L91-tm5-aer 255 91 true tm5 true aer 2010-01-01
T255L91-tm5-none 255 91 true tm5 false - 2010-01-01
T255L91-bascoetm5-aer 255 91 true bascoetm5 true aer 2010-01-01
T255L91-bascoetm5-none 255 91 true bascoetm5 false - 2010-01-01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2 t21test_xios_t255ac
═════════════════════
2.1 Presentation
────────────────
• This is as close as possible to /t21test/, so they could eventually
be combined if desired.
• Short 1-day run to demonstrate the OIFS-AC. It has a very limited
set of options. Note that you cannot increase the runtime of the
'oiac' experiment because the provided ICMCL file (albedos/LAIs
climatology) is for 2013-07-01 to 2013-07-03 only.
• The /setup-exp.sh/ script sets the experiment up and run it.
• Check the "Set platform and read its configuration" section, where
the RUNDIR is defined and the OIFS env variables (same as for
compilation) are retrieved.
• You need a wrapper to run it on your HPC (see examples in the
platform subdir)
• XIOS and Chemistry are optional and can be switched on at the
command line. For more details, read script header:
┌────
│ setup-exp.sh -h
└────
• Possible experiments are:
• 'epc8' on the TF21 grid (same as in t21test dir)
• 'oiac' on the TL255 grid, which uses a modified CB05 chemistry
scheme (referred to as TM5) and the AER aerosol scheme on 60
levels. You can switch on the BASCOE chemistry scheme by editing
the CHEM_SCHEME variable in the /setup-exp.sh/.
2.2 How to run
──────────────
• create a wrapper job script that you will submit to your job
manager. The job script must set OMP_NUM_THREADS, NPROC, and
OIFS_RUNCMD (which probably differs if you are using XIOS or not),
and call /setup-exp.sh/. See examples in:
┌────
│ runtime/platform/*job.tmpl
└────
• create a platform configuration file in:
┌────
│ runtime/platform/oifs-config.${platform}.sh
└────
• set the platform variable in /setup-exp.sh/ (line 89)
• submit
2.3 Tested configurations
─────────────────────────
━━━━━━━━━━━━━━━━━━━━━━━━━━━━
cca-cray
────────────────────────────
t21 ok
t21 + xios ok
tm5/aer ok
tm5/aer + xios ok
━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2.4 Output and diagnostics
──────────────────────────
Diagnostics - At the end of 'oiac' experiment, a couple of budgets are
gathered. These can be compared with previously run results, which are
included in the 'ctrl' directory. For example, if you expect the exact
same result, you could:
┌────
│ EXPID=oiac; RUNDIR=${SCRATCH}/${OIFS_CYCLE}/${EXPID} ; RUN_NUMBER=3 #checkoifs
│ fref1=${OIFS_HOME}/t21test_xios_ac/ctrl/massdia_chem__oiac_2013070100.txt
│ fref2=${OIFS_HOME}/t21test_xios_ac/ctrl/ozonbud_chem__oiac_2013070100.txt
│ fnew1=${RUNDIR}/output${RUN_NUMBER}/massdia_chem_oiac.txt
│ fnew2=${RUNDIR}/output${RUN_NUMBER}/ozonbud_chem_oiac.txt
│ diff $fref1 $fnew1
│ diff $fref2 $fnew2
└────
Output:
• ${RUNDIR}/output${RUN_NUMBER} : contains log, chem diagnostics, and
GG+SH grib files if not using XIOS.
• ${RUNDIR}/xios_output : has netcdf files if using XIOS.
For pointers to handle the output on the reduced Gaussian grid, see
the Output section below.
3 AC-experiments
════════════════
3.1 Overview
────────────
In the AC-experiments directory, you will find:
• low level scripts, in the /scripts/ subdirectory. Several of them
provides a set of functions, like a library.
• two main level scripts, one for pre-processing and one for running
OIFS
• one high level workflow script, which is mainly an example, known to
work at ECMWF.
Here's an overview:
┌────
│ AC-experiments
│ ├── config.h # Config file
│ ├── ctrl
│ │ ├── context_ifs.xml # xios resource
│ │ ├── ifs_xml # xios resource
│ │ ├── iodef.xml # xios resource
│ │ ├── namelist.ifs.sh # script to generate OIFS namelist
│ │ └── Table # dir with AC tracers definition tables
│ │
│ ├── oifs-run.sh # Main script to setup rundir and launch OIFS executable
│ ├── prep-ic-icmcl-compo.sh # Main script to gather IC/ICML/ICMCL-COMPO
│ │
│ ├── platform
│ │ ├── ecmwf-cca.job.tmpl # job template to run without xios (PBSpro, qsub)
│ │ ├── ecmwf-cca+xios.job.tmpl # job template to run with xios (PBSpro, qsub)
│ │ ├── ecmwf-cca-workflow.sh # Example workflow script for run with series of restarts
│ │ ├── knmi-rhino.job.tmpl # job template to run without xios (SLURM)
│ │ ├── oifs-config.ecmwf-cca.sh # platform environment (ECMWF)
│ │ └── oifs-config.knmi-rhino.sh # platform environment (KNMI)
│ │
│ └── scripts # Set of low levels scripts
│ ├── add_nrt_fire_chem
│ ├── gaussgr
│ ├── get_alb_lai_intLimits
│ ├── get_tablecol
│ ├── grib_def.h
│ ├── lib_general.sh # lib of utilities
│ ├── lib_chem_setup.sh # lib of functions to create the AC namelists
│ ├── lib_icmcl_clima # lib of functions to create ICMCL files
│ ├── lib_icmcl_compo # lib of functions to create ICMCL-COMPO files
│ └── lib_initcond.sh # lib of functions to create Initial Conditions (IC) files
└────
3.2 Main scripts
────────────────
There are two main scripts to setup and run an experiment. Although
experiments are configured by editing /config.h/, few script arguments
and options allow for a flexible workflow.
3.2.1 Pre-processing
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
The /prep-ic-icmcl-compo.sh/ script prepares the Initial Conditions,
the climatology, and the atmospheric composition fluxes before you run
OIFS. The main purpose is to create or copy three sets of files (IC,
ICMCL and ICMCL-COMPO) needed to run an experiment. Paths to find
those files are set in the /config.h/. See the section DATASETS in
that file for details. Note that it is possible to use different data
than those provide to test OpenIFS/AC but keep in mind this requires
access to MARS archives at ECMWF.
┌────
│ Usage:
│ prep-ic-icmcl-compo.sh START_DATE END_DATE
│
│ Setup or create in $AC_IC_DIR:
│
│ - Initial Conditions for START_DATE (if not continuing/restarting a run)
│ ICs are looked for in $OIFS_INIDATA_DIR. If not found, they are created.
│
│ - ICMCL climatology file for START_DATE to END_DATE.
│ If $ICMCLDIR dir is set (in config.h), the ICMCL file is linked
│ from $ICMCLDIR/ICMCL-yyyymm0100
│ (Note: can contain data for dates before or after those required)
│
│ - ICMCL-COMPO (Emissions and Depositions) for START_DATE to END_DATE.
│ ICMCL-COMPO are looked for in $OIFS_INIDATA_DIR. If not found, it is created.
│
│ (If \$AC_IC_DIR/ICMCL-INIT-COMPO-... exists, it will not be
│ created again. This allows for faster re-run when developing.)
│
│ START_DATE, END_DATE:
│ - Must be in a format understood by 'date -d' (eg 2010-12-15)
│
└────
Note that:
• If prepared beforehand, ICMCL and ICMCL-COMPO can contain data for dates
before or after those requested, but filename formats expected BY THIS
SCRIPT are:
= ICMCL : ICMCL-yyyymm0100
= ICMCL-COMPO : ICMCL-INIT-COMPO or ICMCL-INIT-COMPO-START_DATE-END_DATE (with dates in YYYYMMDDhh)
• $OIFS_INIDATA_DIR, $ICMCLDIR and $AC_IC_DIR are set in /config.h/
3.2.2 OIFS run
╌╌╌╌╌╌╌╌╌╌╌╌╌╌
The /oifs-run.sh/ script reads the experiment setup (config.h), sets a
rundir up (if not already done) and launch the OIFS executable to
start or continue the experiment.
You will need a wrapper around /oifs-run.sh/ to define
OMP_NUM_THREADS, OIFS_NPROC, and OIFS_RUNCMD, and sent it to your HPC
compute nodes. Some example wrappers, working with and without XIOS:
• AC-experiments/platform/ecmwf-cca.job.tmpl
• AC-experiments/platform/ecmwf-cca+xios.job.tmpl
3.3 Configuration
─────────────────
There is a lot of comments in the /config.h/ to explain what the
different entries are. A few words about the main AC keys:
To activate chemistry, set:
• LCHEM=true
• choose chemistry scheme (tm5 or bascoetm5) and corresponding table
version:
• CHEM_SCHEME = tm5 or bascoetm5
• CHEM_VER = ver15 or ver2f
And to activate aerosols:
• LAERO=true
• if not, LCHEM_AEROI (use MACC to get aerosol scattering/absorption)
must be set to false
• choose aerosol scheme and corresponding table version:
• AERO_SCHEME = aer
• AERO_VER = ver1
3.4 Examples
────────────
3.4.1 Example 1 - A simple run
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
• Configure your experiment by editing /config.h/. To run a 10-day
experiment day, use:
┌────
│ run_start_date="2010-01-01"
│ run_end_date="2010-01-11"
└────
The restart frequency should be longer than the run, eg:
┌────
│ rst_freq="1 month"
└────
• Create or gather ICs:
┌────
│ prep-ic-icmcl-compo.sh 2010-01-01 2010-01-11
└────
It is recommended to run it on a compute node, if the ICMCL-COMPO
has to be created.
• Then submit a job to launch an oifs executable.
┌────
│ {qsub, sbatch,...} wrapper.sh
└────
3.4.2 Example 2 - Longer experiment with stop-and-restart
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
• Define your experiment by editing /config.h/. To run a 2-month
experiment with a stop-and-restart after the 1st month (ie 2
legs/chunks of 1 month), just set:
┌────
│ run_start_date="2010-01-01"
│ run_end_date="${run_start_date} + 2 months"
│ rst_freq="1 month"
│ USE_RESTART=true
└────
• Create or gather ICs for every leg:
┌────
│ prep-ic-icmcl-compo.sh 2010-01-01 2010-02-01
│ prep-ic-icmcl-compo.sh 2010-02-01 2010-03-01
└────
These can be run in parallel. But if ICMCLDIR is not provided, then
ICMCL climatology needs to be created, and you have to wait that the
first job is finished before starting the others (which can then run
in parallel).
• Submit your wrapper-around-oifs-run once:
┌────
│ {qsub, sbatch} wrapper.sh
└────
Wait until it is finished, then resubmit it again to simulate the
second month. No need to edit your /config.h/, it will pickup where
it stopped.
3.4.3 Example 3 - Using a workflow script
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
The 4 jobs in the previous example, can be all triggered automatically
and in a timely fashion with a single script. The
/platform/ecmwf-cca-workflow.sh/ provides an example that works on
cca@ECMWF.
• Define a 2-month experiment with a stop-and-restart after the 1st
month:
┌────
│ run_start_date="2010-01-01"
│ run_end_date="${run_start_date} + 2 months"
│ rst_freq="1 month"
└────
• Create and queue jobs with correct dependencies:
┌────
│ ./workflow-cca.sh
└────
3.4.4 About performance - Long pre-processing
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
ICMCL-COMPO could be prepared for the entire run at once. In the
example 2 above, it would be done with:
┌────
│ prep-ic-icmcl-compo.sh 2010-01-01 2010-03-01
└────
Then you simulate the two legs one after the other, like above. There
is no advantage in doing that. Indeed, the creation of the ICMCL-COMPO
can be quite expensive (~15 min for two days!). Better to limit it to
monthly or daily files that can be stitched together afterwards if needed.
3.5 Creating ICs at the end of run
──────────────────────────────────
When setting LAC4IC=true, an AC-for-IC grib file is created at the end
of a run. It is a snapshot of all AC tracers, which can be used by the
pre-processing script to create Initial Conditions. The main
requirement is:
• no netCDF output (LXIOS=false)
With the AC-for-IC file, you can create IC for a new experiment, which
will continue from the previous experiment as far as AC tracers are
concerned. The meteo fields to complete the IC can come from an
archived experiment or a set of standard meteo IC. This workflow is
desirable when you want to reinitialize your integration with an
updated weather forecast (which does not provide AC tracers) for
example.
3.5.1 Example - A series of daily runs
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
• Configure your 1st day experiment:
┌────
│ run_start_date="2010-01-01"
│ run_end_date="2010-01-02"
└────
With a restart frequency equal or longer than the run, you can
switch off the creation of restart:
┌────
│ rst_freq="1 month"
│ USE_RESTART=false
└────
• Create ICs
┌────
│ prep-ic-icmcl-compo.sh 2010-01-01 2010-01-02
└────
• Then submit a job to launch oifs executable
┌────
│ {qsub, sbatch} wrapper.sh
└────
This will create an extra file we will use in the next day
experiment:
┌────
│ AUXFILE=${RUNDIR}/output/001/ICMUA${EXP}.ac_rst
└────
• Configure your next day experiment, with a new name (or after
removing the previous rundir - after moving away the AUXFILE!):
┌────
│ EXP=bcde
│ run_start_date="2010-01-02"
│ run_end_date="2010-01-03"
└────
In the /config.h/, be sure to:
- set OIFS_INIDATA_DIR to nothing, or comment it.
- set the variable AC_IC_ONCE to the AUXFILE from the previous exp.
- set the MET_IC_DIR or modify the "variables to retrieve METEO initial
conditions from an archived experiment" section as you see fit.
• Create ICs
┌────
│ prep-ic-icmcl-compo.sh 2010-01-02 2010-01-03
└────
• Then submit a job to launch oifs executable
┌────
│ {qsub, sbatch} wrapper.sh
└────
This will create an extra file for another run:
┌────
│ AUXFILE=${RUNDIR}/output/001/ICMUA${EXP}.ac_rst
└────
• Etc.
4 Output
════════
Output data are on the reduced grid - which is an unstructured grid in the
netcdf files. To go further, you may want to remap the output. This can be
achieved with /cdo/. The procedure is slightly different for grib and netCDF
files. Check the scripts in the /postproc/ directory for examples dealing
with grib files. For netCDF files, you must set a grid (grid descriptions
are available in the /ctrl/ directory) before remapping:
┌────
│ cdo -L setgridtype,regular -setgrid,${OIFS_HOME}/t21test_xios_ac/ctrl/TL255-griddes.txt \
│ AMIP_6h_reduced_ml.nc oiac_6h_reduced_ml.nc
└────
Note that available UV-processor output is an experimental product, which is
currently not filled (just zeroes). It concerns two variables: uvp2drad
(gribcode 210055) and toaer340 (gribcode 210056)