Introduction

This section shows how to configure and run OpenIFS and XIOS together.

First of all, it is necessary that OpenIFS is compiled with XIOS (see OpenIFS: Enable XIOS during compilation) and the XIOS executable is available in the experiment directory. OpenIFS should be run as usual (see 6.3 OpenIFS Running a Forecast), but adding the argument '--xios' to oifs_run. This will use by default 1 XIOS server, but in case of needing more servers, it can be done with '--xios --xios-nproc <arg>'.

It is also necessary to enable the 'LXIOS' switch in the 'NAMCT0' namelist (fort.4 file).

Output control: XML files

It is expected that users have knowledge on the use of XIOS which includes the creation of NetCDF files, computation of diagnostics, interpolations, data compression, interpretation of performance parameters, etc. Official documentation such as the user guide or tutorials is available on the webpage: https://forge.ipsl.jussieu.fr/ioserver. However, this section will explain XIOS aspects that are particular to OpenIFS.

OpenIFS and XIOS are controlled using different XML files:

• iodef.xml: it contains basic parameters to control the working mode of XIOS (attached or server mode, 1 or 2 levels of servers, etc), performance parameters, verbose level, etc.
• context_ifs.xml: it contains FullPos parameters to tune the vertical interpolations as well as two optimizations that might improve the execution time.
• axis_def_ifs.xml: it defines the different types of vertical levels available in OpenIFS: model levels, pressure levels, theta levels and PV levels.
• domain_def_ifs.xml: it defines the different domains (or types of grids) available in OpenIFS. By default, data can be output in the native grid (reduced Gaussian) or in a regular Gaussian grid with size 512x256.
• grid_def_ifs.xml: it defines the grids (XIOS terminology) available in OpenIFS used to map fields:
  • Grids for 2D fields are made of a single domain.
  • Grids for 3D fields are made of a domain and an axis.
• field_def_ifs.xml: it defines all the available fields to be output in OpenIFS:
  • Surface fields.
  • 3D fields in model levels.
  • 3D fields in pressure levels.
  • 3D fields in theta levels.
  • 3D fields in PV levels.
  • 3D PEXTRA fields in model levels.
• file_def_ifs.xml: it defines NetCDF files to be written. It allows you to configure many different parameters such as output frequency, compression, diagnostics, output mode (one file vs. multiple files), metadata, timeseries, etc.

Let’s see in more detail how to configure different features particular to OpenIFS-XIOS.

FullPos parameters

In context_ifs.xml different FullPos variables can be tuned. There is a group of variables that control the spectral fitting of FullPos and a second group that contains four different variables named 'NFPCLI', 'LFPQ', 'LTRACEFP' and 'RFPCORR'. More information about all these variables can be found at  How to control OpenIFS output and in the latest FullPos user guide.

Optimizations for sending data from OpenIFS processes to XIOS servers

There are two available optimizations that might be useful to improve the execution time under some circumstances. It is not possible to predict in which conditions, as many factors have to be taken into consideration. For this reason it is necessary to test them. They are disabled by default, but can be enabled in context_ifs.xml:

• 'LOPT_SEND': it enables a mechanism to change where data is sent from XIOS clients (OpenIFS processes) to XIOS servers to truly overlap OpenIFS computations with XIOS communications. 'Changing where data is sent' means to buffer output data and delay its sending until physics tendencies computation which is free of MPI communication. If this optimization is not enabled, output data is sent to XIOS servers immediately when it is produced by FullPos, 'colliding' with OpenIFS synchronous internal communications.
• 'LSINGLE_PREC_SEND': it sends data from XIOS clients to XIOS servers in single precision (32 bits) instead of double precision (64 bits). This allows you to send half of the data to decongest the network.

Define different vertical level values

It is possible to define different vertical levels to be interpolated from model levels in axis_def_ifs.xml. In particular, it is possible to define pressure levels, theta levels and PV levels. For example, if you want to define four theta levels, you should edit 'n_glo' and 'value’:

<axis id="theta_levels" long_name="vertical theta (potential temperature) levels" n_glo="4" value="(0,3)[300 320 340 360]" />

In addition, it is possible to output fields with a subset of the vertical levels. This XIOS feature is known as zoom and you have to specify the indexes (from the original axis) of the levels that you want to output, and NOT the level values themselves. From the previous example, if you are only interested in theta levels 320 and 360, you should edit 'index' as follows:

<zoom_axis index="(0,1)[1 3]" />

Note that indexes go from '0' to 'n_glo - 1'.

Define different regular Gaussian domains

Define different grids from new axes and domains

Enable regular lat-lon grid output

Understanding output frequency, sampling frequency, NFRHIS and NFRPOS

Computational performance considerations

This section reviews different aspects that can considerably affect the computational performance of OpenIFS-XIOS and consequently increase the total execution time.

One file vs. multiple file mode

Horizontal interpolations

Not correctly setting freq_op, NFRHIS and NFRPOS

Spectral transformations

Lustre filesystem