Note |
---|
Note that this documentation applies to the OpenIFS/AC code extension for OpenIFS 43r3. This does not apply to users of OpenIFS 48r1 or later model releases. |
...
Table of Contents |
---|
...
This page describes how OpenIFS/AC v1 (derived from OpenIFS 43r3v2) can be obtained, built and used for forecast experiments. The documentation shows examples of how to use the model on the ECMWF Atos Sequana XH2000 HPCF (hpc2020). Adjustments will need to be made for local installations.
Info |
---|
Enquiries about OpenIFS/AC or technical questions should in the first instance be sent to openifs-support@ecmwf.int |
OpenIFS/AC compared to the standard OpenIFS model
...
The standard OpenIFS model is described in detail in the OpenIFS 43r3 User Guide. The user guide introduces the model's components and supported model grids. It informs about additional required software packages and how to build the OpenIFS model. It further describes how to carry out an acceptance test after a successful model build in order to verify the model's functionality.
Info |
---|
Users who are new to OpenIFS should first read the OpenIFS 43r3 User Guide's sections 1 to 5 before proceeding with OpenIFS/AC, as the steps described therein are required for building OpenIFS/AC. The instructions in the user guide will not be repeated here. |
...
No Format |
---|
ftp ftp.ecmwf.int (login with username and password for OpenIFS users) cd src/openifs-ac get openifs-ac.43r3.v1.tgzgz bye |
Installing the OpenIFS/AC sources
...
- Create a new home for the OpenIFS/AC installation. Here we call this location /home/openifs-ac.
- Change into your new directory and unpack therein the original OpenIFS model sources: tar xvzf oifs43r3v2.tgz
- In a second step, in the same folder unpack gunzip the OpenIFS/AC sources : tar xvzf and patch the original OpenIFS:
- gunzip openifs-ac.43r3.v1
- .gz
- patch -p1 <openifs-ac.43r3.v1
Info |
---|
Note that some files of the original model will be replacedmodified. It is therefore essential to install the original OpenIFS 43r3v2 sources first before installing the OpenIFS/AC sources. |
...
- Change into your $OIFS_HOME directory where OpenIFS is installed: cd $OIFS_HOME
- In this directory extract the downloaded source package: tar xvzf
- gunzip openifs-ac.43r3.v1
- .gz
- patch -p1 <openifs-ac.43r3.v1
Info |
---|
Note that this unpacking will not only add new files to your existing $OIFS_HOME but it will also overwrite some of the existing files with updated versions required for OpenIFS/AC. |
...
- To do this we run the standard t21test for experiment epc8 (T21L19) with the
setup-exp.sh
new script. - The name for the hardware platform needs to be set (l.89, e.g.
platform=ecmwf-hpc2020
) - The test experiment runs in the
RUNDIR
test experiment runs in theRUNDIR
directory specified insetup-exp.sh
(l.94, the default is$SCRATCH/43r3/epc8/
). - We also need to use the command line option -x and give as argument the absolute full path to the
master.exe
binary executable. - In
setup-exp.sh
you should change the value forRUN_NUMBER
otherwise any pre-existing model output from this test will be overwritten.
No Format |
---|
./setup-exp.sh -n 16 -t 2 -x /home/damk/oifs/model/openifs-ac/make/gnu-opt-chem-fftw/oifs/bin/master.exe |
...
/gnu-opt-chem-fftw/oifs/bin/master.exe |
Test result: As before an error calculation file res_021_0072
is created in the run directory and the model output is found in the subfolder output1
(this name depends on the RUN_NUMBER variable).
Info |
---|
Running on the ECMWF HPCF: |
With chemistry: Now repeat the test with enabled model chemistry. This is a short 1-day test run that uses the newly added source code of OpenIFS/AC. It has a very limited set of options and should not be used for forecast experiments. Note that you cannot increase the experiment duration beyond one day because the boundary conditions are only supplied from 2013-07-01 to 2013-07-03.
...
No Format |
---|
AC-experiments/ ├── config.h # MAIN experiment config file ├── ctrl/ │ ├── context_ifs.xml # xios resource │ ├── ifs_xml/ # xios resource │ ├── iodef.xml # xios resource │ ├── namelistfc │ ├── namelist.ifs-epc8.sh # namelist for t21test without chemistry │ ├── namelist.ifs-oiac.sh # namelist for t21test with chemistry │ ├── namelist.ifs.sh # namelist template for forecast experiments │ ├── Table/ # dir with AC tracers │ └── wam_namelist ├── ecmwf-atoshpc2020.job # job wrapper script for oifs-run.sh ├── oifs-run.sh # MAIN run script ├── platform/ # dir with platform-specific templates ├── prep_emis/ # dir with emission utilities ├── prep-ic-icmcl-compo.sh # MAIN script to prepare initial/boundary conditions └── scripts/ # dir with low-level scripts ├── add_nrt_fire_chem ├── gaussgr ├── get_alb_lai_intLimits ├── get_tablecol ├── grib_def.h ├── lib_chem_setup.sh ├── lib_general.sh # lib of utilities ├── lib_icmcl_clima.sh # lib of functs to create ICMCL boundary cond. files ├── lib_icmcl_compo.sh # lib of functs to create ICMCL-COMPO files ├── lib_initcond_offline.sh └── lib_initcond.sh # lib of functs to create initial conditions files |
...
- prep-ic-icmcl-compo.sh - a script to generate initial conditions for chemical fields for the selected run time. Begin and end date need to be given as an argument.
- config.h - settings for model installation, paths, and experiment configuration.
- oifs-run.sh - this script controls the workflow for the experiment: reads experiment config, sets up run directories (in $SCRATCH), loads platform configuration, edits namelists, links to initial conditions and executes OIFS_RUNCMD
- ecmwf-atoshpc2020.job wrapper script for oifs-run.sh, required settings for the batch scheduler
...
- it is necessary to create a platform-specific machine config file: oifs-config.ecmwf-atoshpc2020.sh
The above ecmwf-atoshpcs2020.job wrapper script to submit the batch job is based on templates in the platform directory; a number of templates exist and for the ECMWF HPCF the following templates are relevant:
...
No Format |
---|
PLATFORM='ecmwf-atoshpc2020' # this needs to match the name of the platform configuration file EXP=abcd # you should select here an experiment ID (4-letter alphanumeric word, must not start with '0') RUNDIR=${SCRATCH}/43r3/experiments/${EXP}/runs # this is the experiment directory AC_IC_DIR=${SCRATCH}/43r3/experiments/${EXP}/chem_ic # this is the stageingstaging directory for initial and boundary conditions data files # Simulation start and end date. If setting time (not mandatory), **put it before** the date run_start_date="00:00 2010-01-01" run_end_date="${run_start_date} + 2 days" rst_freq='6 day' # Run for 6 days without restarts. Our experiment will finish after 2 days, so this will be ignored. USE_RESTART=false # Do not write restart files. |
...
Code Block |
---|
PLATFORM='ecmwf-atoshpc2020' RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs' # where your experiment will run AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic' # where your chemical initial conditions will be created run_end_date="2010-01-03" rst_freq='6 day' USE_RESTART=false LCHEM=true CHEM_SCHEME="tm5" CHEM_VER="ver15" OIFS_INIDATA_DIR='${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-tm5-none' LAERO=false LAERCHEM=false LCHEM_AEROI=false LINJ_CHEM=false LINJ_AER=false LAEROSFC=false LWAM=true LRLXG=false |
...
Code Block |
---|
PLATFORM='ecmwf-atoshpc2020' RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs' # where your experiment will run AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic' # where your chemical initial conditions will be created run_end_date="2010-01-03" rst_freq='6 day' USE_RESTART=false LCHEM=true CHEM_SCHEME="tm5" CHEM_VER="ver15" OIFS_INIDATA_DIR='${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-tm5-aer' LAERO=true LAERCHEM=true LCHEM_AEROI=true LINJ_CHEM=true LINJ_AER=true LAEROSFC=true LWAM=true LRLXG=false |
...
Code Block |
---|
PLATFORM='ecmwf-atoshpc2020' RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs' # where your experiment will run AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic' # where your chemical initial conditions will be created run_end_date="2010-01-03" rst_freq='6 day' USE_RESTART=false LCHEM=true CHEM_SCHEME="bascoetm5" CHEM_VER="ver2f" OIFS_INIDATA_DIR='${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-bascoetm5-none' LAERO=false LAERCHEM=false LCHEM_AEROI=false LINJ_CHEM=false LINJ_AER=false LAEROSFC=false LWAM=true LRLXG=false |
...
Code Block |
---|
PLATFORM='ecmwf-atoshpc2020' RUNDIR='${SCRATCH}/43r3/experiments/${EXP}/runs' # where your experiment will run AC_IC_DIR='${SCRATCH}/43r3/experiments/${EXP}/chem_ic' # where your chemical initial conditions will be created run_end_date="2010-01-03" rst_freq='6 day' USE_RESTART=false LCHEM=true CHEM_SCHEME="bascoetm5" CHEM_VER="ver2f" OIFS_INIDATA_DIR='${OIFS_DATA_DIR}/INITIAL-CONDITIONS/T255L91-bascoetm5-none' LAERO=true LAERCHEM=true LCHEM_AEROI=true LINJ_CHEM=true LINJ_AER=true LAEROSFC=true LWAM=true LRLXG=false |
...
Info |
---|
Note: This option is at an experimental development stage and it is therefore currently not supported for general users. Its implementation depends on scripting that is dependant on the hardware platform. Suggestions for script templates that are used on the ECMWF HPCF can be provided on request. |
There are several reasons why a user may wish to restart a forecast experiment. The increase in computational costs for simulating atmospheric composition may require to break a longer forecast experiment into individual sections ("legs"). Also, a restart will allow to initialise the meteorology in the model with data taken from an analysis of the current modelled time.
Due to the time scales associated with the different chemical processes and the lifetime of chemical tracers with respect to their loss processes, it is often convenient to retain the chemical tracer fields in the model simulation and update only the driving meteorology at the time of the restart. In some applications, the model could even be initialised every single day with an up-to-date meteorological analysis field while the chemical fields are left to evolve. The previous section described how chemical initial conditions can be created at the end of an experiment run, thus retaining the chemical tracer information from the previous experiment leg.
We will here provide some examples of the principle steps required for restarting experiments.
Example 1: Longer experiment with manual stop and restart
The experiment parameters need to be defined by editing config.h
. The following settings are used for a 2-month experiment with a stop-and-restart after the first month. This will result in two experiment legs of 1 month duration each.
No Format |
---|
run_start_date="2010-01-01" run_end_date="${run_start_date} + 2 months" rst_freq="1 month" USE_RESTART=true |
The initial conditions for each leg need to be either created or gathered from a prepared repository.
No Format |
---|
prep-ic-icmcl-compo.sh 2010-01-01 2010-02-01 prep-ic-icmcl-compo.sh 2010-02-01 2010-03-01 |
These processes can be run
...
simultaneously. The meteorological fields and initial conditions can be requested from the OpenIFS Data Hub. The wrapper script needs to be submitted for the first experiment leg. Once the experiment has completed, config.h needs to be edited to use the initial experiment data from the date and time when the first leg has stopped. Then the wrapper script for the second month should be submitted.
Example 2: Using a workflow script
The 4 jobs in the previous example, can be all triggered automatically and in a timely fashion with a single script. The $OIFS_HOME/AC-experiments/platform/<platform-name>-workflow.sh
templates provide examples that need to be adjusted for the local hardware and file paths. In this case the config.h
file only needs editing once and instead of the batch job wrapper script the workflow script is submitted.