...
Multiple forecasts with restarts
unclear how to do thatThis option is currently not supported generally as it depends on scripting that is dependant on the computing system used. Suggestions for a version that works on the Atos can be provided upon request. 
README
━━━━━━━━━━━━━━━━━━━━━━━━━
ATMOSPHERIC COMPOSITION
━━━━━━━━━━━━━━━━━━━━━━━━━
Table of Contents
─────────────────
1. Test data
.. 1. Datasets for OIFS/AC
.. 2. Overview table of provided Initial Conditions
2. t21test_xios_t255ac
.. 1. Presentation
.. 2. How to run
.. 3. Tested configurations
.. 4. Output and diagnostics
3. AC-experiments
.. 1. Overview
.. 2. Main scripts
..... 1. Pre-processing
..... 2. OIFS run
.. 3. Configuration
.. 4. Examples
..... 1. Example 1 - A simple run
..... 2. Example 2 - Longer experiment with stop-and-restart
..... 3. Example 3 - Using a workflow script
..... 4. About performance - Long pre-processing
.. 5. Creating ICs at the end of run
..... 1. Example - A series of daily runs
4. Output
5. Using the workflow without restart
━━━━━━━━━━━━━━━━━━━━━━━━━
There are two of directories for the Atmospheric Composition work. The
/t21test_xios_t255ac/ is just a test run with almost no options that you
can set. In /AC-experiments/ you can configure your experiment in
greater detail. In all cases, you need to install some additional data
for the AC work.
1 Test data
═══════════
1.1 Datasets for OIFS/AC
────────────────────────
• You can fetch the main data dedicated to AC configurations from ECFS:
┌────
│ ec:/nm6/oifs/oifs-ac-43r3-chem.tgz
│ ec:/nm6/oifs/oifs-ac-43r3-xtra.tgz
└────
This will unpack to three directories: 43r3/chemistry, 43r3/climate.v015,
and '43r3/xdata'. The second one adds input to the standard
'ifsdata_cy43r3_climate.v015_255l.tgz'
• Unpack them in the same directory where you unpack the default
OpenIFS data, which should then contain:
┌────
│ 43r3/chemistry # AC specific
│ 43r3/xdata # AC specific
│ 43r3/rtables # Standard dataset for OIFS-43r3
│ 43r3/climate.v015 # Standard dataset for OIFS-43r3 + AC specifics
│ 43r3/ifsdata # Standard dataset for OIFS-43r3
│ 43r3/wam # Standard dataset for OIFS-43r3
└────
• A set of initial conditions is available:
┌────
│ ec:/nm6/oifs/oifs-ac-43r3-initcond.tgz
└────
Unpack into a directory: INITIAL-CONDITIONS
• A set of ICMCL climatology for the years 2010 on the TL255L90 grid (kindly
prepared by ECMWF) and few days of 2013 on TL255L60 is also needed:
┌────
│ ec:/nm6/oifs/ICMCL-2010.tgz
└────
Unpack into a directory: ICMCL
1.2 Overview table of provided Initial Conditions
─────────────────────────────────────────────────
The tested and supported configurations are:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
RESOL LEVELS LCHEM CHEM_SCHEME LAERO AERO_SCHEME START_DATE
───────────────────────────────────────────────────────────────────────────────────────────
T255L60-bascoetm5-aer 255 60 true bascoetm5 true aer 2013-07-01
T255L60-tm5-aer 255 60 true tm5 true aer 2013-07-01
───────────────────────────────────────────────────────────────────────────────────────────
T255L91-tm5-aer 255 91 true tm5 true aer 2010-01-01
T255L91-tm5-none 255 91 true tm5 false - 2010-01-01
T255L91-bascoetm5-aer 255 91 true bascoetm5 true aer 2010-01-01
T255L91-bascoetm5-none 255 91 true bascoetm5 false - 2010-01-01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2 t21test_xios_t255ac
═════════════════════
2.1 Presentation
────────────────
• This is as close as possible to /t21test/, so they could eventually
be combined if desired.
• Short 1-day run to demonstrate the OIFS-AC. It has a very limited
set of options. Note that you cannot increase the runtime of the
'oiac' experiment because the provided ICMCL file (albedos/LAIs
climatology) is for 2013-07-01 to 2013-07-03 only.
• The /setup-exp.sh/ script sets the experiment up and run it.
• Check the "Set platform and read its configuration" section, where
the RUNDIR is defined and the OIFS env variables (same as for
compilation) are retrieved.
• You need a wrapper to run it on your HPC (see examples in the
platform subdir)
• XIOS and Chemistry are optional and can be switched on at the
command line. For more details, read script header:
┌────
│ setup-exp.sh -h
└────
• Possible experiments are:
• 'epc8' on the TF21 grid (same as in t21test dir)
• 'oiac' on the TL255 grid, which uses a modified CB05 chemistry
scheme (referred to as TM5) and the AER aerosol scheme on 60
levels. You can switch on the BASCOE chemistry scheme by editing
the CHEM_SCHEME variable in the /setup-exp.sh/.
2.2 How to run
──────────────
• create a wrapper job script that you will submit to your job
manager. The job script must set OMP_NUM_THREADS, NPROC, and
OIFS_RUNCMD (which probably differs if you are using XIOS or not),
and call /setup-exp.sh/. See examples in:
┌────
│ runtime/platform/*job.tmpl
└────
• create a platform configuration file in:
┌────
│ runtime/platform/oifs-config.${platform}.sh
└────
• set the platform variable in /setup-exp.sh/ (line 89)
• submit
2.3 Tested configurations
─────────────────────────
━━━━━━━━━━━━━━━━━━━━━━━━━━━━
cca-cray
────────────────────────────
t21 ok
t21 + xios ok
tm5/aer ok
tm5/aer + xios ok
━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2.4 Output and diagnostics
──────────────────────────
Diagnostics - At the end of 'oiac' experiment, a couple of budgets are
gathered. These can be compared with previously run results, which are
included in the 'ctrl' directory. For example, if you expect the exact
same result, you could:
┌────
│ EXPID=oiac; RUNDIR=${SCRATCH}/${OIFS_CYCLE}/${EXPID} ; RUN_NUMBER=3 #checkoifs
│ fref1=${OIFS_HOME}/t21test_xios_ac/ctrl/massdia_chem__oiac_2013070100.txt
│ fref2=${OIFS_HOME}/t21test_xios_ac/ctrl/ozonbud_chem__oiac_2013070100.txt
│ fnew1=${RUNDIR}/output${RUN_NUMBER}/massdia_chem_oiac.txt
│ fnew2=${RUNDIR}/output${RUN_NUMBER}/ozonbud_chem_oiac.txt
│ diff $fref1 $fnew1
│ diff $fref2 $fnew2
└────
Output:
• ${RUNDIR}/output${RUN_NUMBER} : contains log, chem diagnostics, and
GG+SH grib files if not using XIOS.
• ${RUNDIR}/xios_output : has netcdf files if using XIOS.
For pointers to handle the output on the reduced Gaussian grid, see
the Output section below.
3 AC-experiments
════════════════
3.1 Overview
────────────
In the AC-experiments directory, you will find:
• low level scripts, in the /scripts/ subdirectory. Several of them
provides a set of functions, like a library.
• two main level scripts, one for pre-processing and one for running
OIFS
• one high level workflow script, which is mainly an example, known to
work at ECMWF.
Here's an overview:
┌────
│ AC-experiments
│ ├── config.h # Config file
│ ├── ctrl
│ │ ├── context_ifs.xml # xios resource
│ │ ├── ifs_xml # xios resource
│ │ ├── iodef.xml # xios resource
│ │ ├── namelist.ifs.sh # script to generate OIFS namelist
│ │ └── Table # dir with AC tracers definition tables
│ │
│ ├── oifs-run.sh # Main script to setup rundir and launch OIFS executable
│ ├── prep-ic-icmcl-compo.sh # Main script to gather IC/ICML/ICMCL-COMPO
│ │
│ ├── platform
│ │ ├── ecmwf-cca.job.tmpl # job template to run without xios (PBSpro, qsub)
│ │ ├── ecmwf-cca+xios.job.tmpl # job template to run with xios (PBSpro, qsub)
│ │ ├── ecmwf-cca-workflow.sh # Example workflow script for run with series of restarts
│ │ ├── knmi-rhino.job.tmpl # job template to run without xios (SLURM)
│ │ ├── oifs-config.ecmwf-cca.sh # platform environment (ECMWF)
│ │ └── oifs-config.knmi-rhino.sh # platform environment (KNMI)
│ │
│ └── scripts # Set of low levels scripts
│ ├── add_nrt_fire_chem
│ ├── gaussgr
│ ├── get_alb_lai_intLimits
│ ├── get_tablecol
│ ├── grib_def.h
│ ├── lib_general.sh # lib of utilities
│ ├── lib_chem_setup.sh # lib of functions to create the AC namelists
│ ├── lib_icmcl_clima # lib of functions to create ICMCL files
│ ├── lib_icmcl_compo # lib of functions to create ICMCL-COMPO files
│ └── lib_initcond.sh # lib of functions to create Initial Conditions (IC) files
└────
3.2 Main scripts
────────────────
There are two main scripts to setup and run an experiment. Although
experiments are configured by editing /config.h/, few script arguments
and options allow for a flexible workflow.
3.2.1 Pre-processing
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
The /prep-ic-icmcl-compo.sh/ script prepares the Initial Conditions,
the climatology, and the atmospheric composition fluxes before you run
OIFS. The main purpose is to create or copy three sets of files (IC,
ICMCL and ICMCL-COMPO) needed to run an experiment. Paths to find
those files are set in the /config.h/. See the section DATASETS in
that file for details. Note that it is possible to use different data
than those provide to test OpenIFS/AC but keep in mind this requires
access to MARS archives at ECMWF.
┌────
│ Usage:
│ prep-ic-icmcl-compo.sh START_DATE END_DATE
│
│ Setup or create in $AC_IC_DIR:
│
│ - Initial Conditions for START_DATE (if not continuing/restarting a run)
│ ICs are looked for in $OIFS_INIDATA_DIR. If not found, they are created.
│
│ - ICMCL climatology file for START_DATE to END_DATE.
│ If $ICMCLDIR dir is set (in config.h), the ICMCL file is linked
│ from $ICMCLDIR/ICMCL-yyyymm0100
│ (Note: can contain data for dates before or after those required)
│
│ - ICMCL-COMPO (Emissions and Depositions) for START_DATE to END_DATE.
│ ICMCL-COMPO are looked for in $OIFS_INIDATA_DIR. If not found, it is created.
│
│ (If \$AC_IC_DIR/ICMCL-INIT-COMPO-... exists, it will not be
│ created again. This allows for faster re-run when developing.)
│
│ START_DATE, END_DATE:
│ - Must be in a format understood by 'date -d' (eg 2010-12-15)
│
└────
Note that:
• If prepared beforehand, ICMCL and ICMCL-COMPO can contain data for dates
before or after those requested, but filename formats expected BY THIS
SCRIPT are:
= ICMCL : ICMCL-yyyymm0100
= ICMCL-COMPO : ICMCL-INIT-COMPO or ICMCL-INIT-COMPO-START_DATE-END_DATE (with dates in YYYYMMDDhh)
• $OIFS_INIDATA_DIR, $ICMCLDIR and $AC_IC_DIR are set in /config.h/
3.2.2 OIFS run
╌╌╌╌╌╌╌╌╌╌╌╌╌╌
The /oifs-run.sh/ script reads the experiment setup (config.h), sets a
rundir up (if not already done) and launch the OIFS executable to
start or continue the experiment.
You will need a wrapper around /oifs-run.sh/ to define
OMP_NUM_THREADS, OIFS_NPROC, and OIFS_RUNCMD, and sent it to your HPC
compute nodes. Some example wrappers, working with and without XIOS:
• AC-experiments/platform/ecmwf-cca.job.tmpl
• AC-experiments/platform/ecmwf-cca+xios.job.tmpl
3.3 Configuration
─────────────────
There is a lot of comments in the /config.h/ to explain what the
different entries are. A few words about the main AC keys:
To activate chemistry, set:
• LCHEM=true
• choose chemistry scheme (tm5 or bascoetm5) and corresponding table
version:
• CHEM_SCHEME = tm5 or bascoetm5
• CHEM_VER = ver15 or ver2f
And to activate aerosols:
• LAERO=true
• if not, LCHEM_AEROI (use MACC to get aerosol scattering/absorption)
must be set to false
• choose aerosol scheme and corresponding table version:
• AERO_SCHEME = aer
• AERO_VER = ver1
3.4 Examples
────────────
3.4.1 Example 1 - A simple run
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
• Configure your experiment by editing /config.h/. To run a 10-day
experiment day, use:
┌────
│ run_start_date="2010-01-01"
│ run_end_date="2010-01-11"
└────
The restart frequency should be longer than the run, eg:
┌────
│ rst_freq="1 month"
└────
• Create or gather ICs:
┌────
│ prep-ic-icmcl-compo.sh 2010-01-01 2010-01-11
└────
It is recommended to run it on a compute node, if the ICMCL-COMPO
has to be created.
• Then submit a job to launch an oifs executable.
┌────
│ {qsub, sbatch,...} wrapper.sh
└────
3.4.2 Example 2 - Longer experiment with stop-and-restart
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
• Define your experiment by editing /config.h/. To run a 2-month
experiment with a stop-and-restart after the 1st month (ie 2
legs/chunks of 1 month), just set:
┌────
│ run_start_date="2010-01-01"
│ run_end_date="${run_start_date} + 2 months"
│ rst_freq="1 month"
│ USE_RESTART=true
└────
• Create or gather ICs for every leg:
┌────
│ prep-ic-icmcl-compo.sh 2010-01-01 2010-02-01
│ prep-ic-icmcl-compo.sh 2010-02-01 2010-03-01
└────
These can be run in parallel. But if ICMCLDIR is not provided, then
ICMCL climatology needs to be created, and you have to wait that the
first job is finished before starting the others (which can then run
in parallel).
• Submit your wrapper-around-oifs-run once:
┌────
│ {qsub, sbatch} wrapper.sh
└────
Wait until it is finished, then resubmit it again to simulate the
second month. No need to edit your /config.h/, it will pickup where
it stopped.
3.4.3 Example 3 - Using a workflow script
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
The 4 jobs in the previous example, can be all triggered automatically
and in a timely fashion with a single script. The
/platform/ecmwf-cca-workflow.sh/ provides an example that works on
cca@ECMWF.
• Define a 2-month experiment with a stop-and-restart after the 1st
month:
┌────
│ run_start_date="2010-01-01"
│ run_end_date="${run_start_date} + 2 months"
│ rst_freq="1 month"
└────
• Create and queue jobs with correct dependencies:
┌────
│ ./workflow-cca.sh
└────
3.4.4 About performance - Long pre-processing
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
ICMCL-COMPO could be prepared for the entire run at once. In the
example 2 above, it would be done with:
┌────
│ prep-ic-icmcl-compo.sh 2010-01-01 2010-03-01
└────
Then you simulate the two legs one after the other, like above. There
is no advantage in doing that. Indeed, the creation of the ICMCL-COMPO
can be quite expensive (~15 min for two days!). Better to limit it to
monthly or daily files that can be stitched together afterwards if needed.
3.5 Creating ICs at the end of run
──────────────────────────────────
When setting LAC4IC=true, an AC-for-IC grib file is created at the end
of a run. It is a snapshot of all AC tracers, which can be used by the
pre-processing script to create Initial Conditions. The main
requirement is:
• no netCDF output (LXIOS=false)
With the AC-for-IC file, you can create IC for a new experiment, which
will continue from the previous experiment as far as AC tracers are
concerned. The meteo fields to complete the IC can come from an
archived experiment or a set of standard meteo IC. This workflow is
desirable when you want to reinitialize your integration with an
updated weather forecast (which does not provide AC tracers) for
example.
3.5.1 Example - A series of daily runs
╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌
• Configure your 1st day experiment:
┌────
│ run_start_date="2010-01-01"
│ run_end_date="2010-01-02"
└────
With a restart frequency equal or longer than the run, you can
switch off the creation of restart:
┌────
│ rst_freq="1 month"
│ USE_RESTART=false
└────
• Create ICs
┌────
│ prep-ic-icmcl-compo.sh 2010-01-01 2010-01-02
└────
• Then submit a job to launch oifs executable
┌────
│ {qsub, sbatch} wrapper.sh
└────
This will create an extra file we will use in the next day
experiment:
┌────
│ AUXFILE=${RUNDIR}/output/001/ICMUA${EXP}.ac_rst
└────
• Configure your next day experiment, with a new name (or after
removing the previous rundir - after moving away the AUXFILE!):
┌────
│ EXP=bcde
│ run_start_date="2010-01-02"
│ run_end_date="2010-01-03"
└────
In the /config.h/, be sure to:
- set OIFS_INIDATA_DIR to nothing, or comment it.
- set the variable AC_IC_ONCE to the AUXFILE from the previous exp.
- set the MET_IC_DIR or modify the "variables to retrieve METEO initial
conditions from an archived experiment" section as you see fit.
• Create ICs
┌────
│ prep-ic-icmcl-compo.sh 2010-01-02 2010-01-03
└────
• Then submit a job to launch oifs executable
┌────
│ {qsub, sbatch} wrapper.sh
└────
This will create an extra file for another run:
┌────
│ AUXFILE=${RUNDIR}/output/001/ICMUA${EXP}.ac_rst
└────
• Etc.
4 Output
════════
Output data are on the reduced grid - which is an unstructured grid in the
netcdf files. To go further, you may want to remap the output. This can be
achieved with /cdo/. The procedure is slightly different for grib and netCDF
files. Check the scripts in the /postproc/ directory for examples dealing
with grib files. For netCDF files, you must set a grid (grid descriptions
are available in the /ctrl/ directory) before remapping:
┌────
│ cdo -L setgridtype,regular -setgrid,${OIFS_HOME}/t21test_xios_ac/ctrl/TL255-griddes.txt \
│ AMIP_6h_reduced_ml.nc oiac_6h_reduced_ml.nc
└────
Note that available UV-processor output is an experimental product, which is
currently not filled (just zeroes). It concerns two variables: uvp2drad
(gribcode 210055) and toaer340 (gribcode 210056)
...