Set compilation environment

If your system uses the 'module' command to load software such as compilers, make sure the required modules are loaded before attempting to build OpenIFS.

Configuring OpenIFS: oifs-config

If this is the first time compiling OpenIFS, the first step is to setup the environment for compilation and selecting OpenIFS configuration options. A successful compilation relies on correctly specifying the location of some system and user libraries (e.g. netCDF and ecCodes).

Find the file oifs-config.editme.sh and make a copy to edit using your favourite editor:

% cd oifs43r3v1
% cp oifs-config.editme.sh  oifs-config.sh
#  .. then edit oifs-config.sh 

OpenIFS configuration options

There are number of variables defined in the oifs-config.sh file that control the behaviour and capabilities of OpenIFS.  The most commonly altered ones are listed in the oifs-config.sh file.

Edit the oifs-config.sh file and set your required options. Most of the options can be left to their defaults.

export OIFS_HOME=<set this to the top level directory of your OpenIFS installation>
export OIFS_DATA_DIR=<set this to the top level directory of the previous downloaded 'ifsdata' files>
export OIFS_LAPACK=<system or openifs>  # default is OpenIFS will use LAPACK library available on your system. If not available set to openifs to use provided code.
export OIFS_FFTW=<disable or enable>    # FFTW is a FFT library that improves performance. Set to enable if it's available on your system.
export OIFS_XIOS=<disable or enable>    # XIOS is a parallel I/O library that provides netCDF output. Set to enable if available and you intend to use it.

OpenIFS compilation options

The key variables in oifs-config.sh to set that affect compilation are (see below for more information):

export OIFS_GRIB_DIR="<set this to the location of your ecCodes installation>"
export OIFS_NETCDF_DIR="<set this to the directory containing the netcdf installation>"
export OIFS_COMP=gnu                      #  this chooses the compiler, either: gnu, intel, or cce (Cray compiler)
export OIFS_BUILD=opt                  #  this chooses the type of build, either 'opt' for optimized, or 'noopt' for un-optimized

See the comments in the 'oifs-config.sh' file for more information about the variables.

Set environment options for compilation

Once you have made desired edits to the oifs-config file, 'source' it to set the environment variables:

% source ./oifs-config.me.sh
Environment for OpenIFS: CY43R3
OpenIFS home directory: /home/users/auser
OpenIFS top-level data directory: /home/users/auser/ecmwf/ifsdata
OpenIFS initial data directory : <editme>
OpenIFS GRIB directory: /home/users/auser/ecmwf/eccodes
OpenIFS netCDF directory: /usr/local/lib64
OpenIFS default compiler and build set to: gnu opt
Added OpenIFS FCM and GRIB to user path
Creating command aliases for OpenIFS: omake, omakenew, oenv

Subsequent compilations

With the oifs-config.sh file correctly set, always 'source' this file to set the environment variables before any compilation of OpenIFS:

% source ./oifs-config.sh

If you prefer not to use the oifs-config.sh, we suggest editing the oifs.fcm directly or making a copy and use that on the fcm command line instead.

FCM

OpenIFS uses software called 'FCM' for the compilation of the code. FCM is contained in the OpenIFS source tarball downloaded from the ftp site.

If you are not using the oifs-config.sh file to set the environment, before trying to compile, make sure the FCM directory is added to your PATH environment variable so the fcm command can be found (if you already have fcm available on your system, we still recommend using the distributed version as this is tested and known to work):

export PATH=$PATH:$HOME/ecmwf/oifs/fcm/bin

If you are not using oifs-config.sh, you will also need to make sure the required OIFS_* environment variables to locate the GRIB and netCDF libraries are set otherwise the compilation will fail.

Build configuration

This is a brief introduction on compiling the OpenIFS model. For a more detailed description of the build system for OpenIFS and the different ways to configure it, see Customizing the OpenIFS build.

Before compiling the model, the build configuration must first be set by several environment variables:

OIFS_COMP - This sets the choice of compiler. The default is 'gnu' which means the gfortran/gcc compilers will be used.  Other choices are 'intel' and 'cce' (Cray compiler).

OIFS_BUILD - This sets the type of build.

These environment variables directly correspond to the names of the FCM configuration files in the make/fcm/ directory in the source. Please see this directory for the choices provided.

Build types: OIFS_BUILD sets the type of build and there will be one FCM configuration file for each type. Build types provided are:

• opt - Recommended optimized compile settings for this compiler. Tested to provide best performance for this platform & compiler.
• noopt - Debugging options. No optimization set for all of the code. Suitable for use with a debugger. OpenMP is disabled.
• nansC - Debugging options with extra checks. Array bound checking enabled and variables initialized to NaN (not-a-number). Use of a variable set to NaN will trigger a floating point exception.  This option is useful for debugging purposes but will run the slowest. It may not work in all cases.

e.g.

export OIFS_COMP=intel
export OIFS_BUILD=opt

means use the Intel ifort compiler and the model will be compiled with full optimization compiler settings. So the FCM build system will expect to find a file:

make/fcm/intel-opt.fcm

Setting instead:

export OIFS_COMP=gnu
export OIFS_BUILD=noopt

means FCM will expect to find a file:

make/fcm/gnu-noopt.fcm

If using GNU on a Mac, please see MacOS X known issues.

Building for other systems

As the environment variables refer to the corresponding filename in the make/fcm  directory it is straightforward to create a build configuration for other environments. Take a copy of an existing file and modify it as necessary. It's also possible to alter compile options by the use of additional environment variables (see below). Which approach you use depends on your personal preference.

Be aware however that the optimization compiler flags in the provided configuration files are the recommended ones. It is impossible to test all combinations of compiler flags so the model may become unstable, results differ or performance may reduce if higher optimizations are tried.

Setting the location of libraries

The directory paths to the libraries OpenIFS needs for compilation are normally set in the 'oifs-config.sh' file (as described above).  If you choose not to use 'oifs-config.sh', you'll need to set the environment variables that the FCM configuration files expect to be defined.

The model requires a number of libraries (see 5.2 Prerequisites) to compile: the GRIB library (ECMWF's ecCodes), the netCDF (43r3+ only), and the LAPACK & BLAS numerical libraries are required. An MPI library is also needed but this is assumed to be provided by compiler wrappers in the MPI implementation (e.g. the command mpif90 rather than gfortran).

To set the location of the GRIB, netCDF & LAPACK libraries, other environment variables need to be defined as the defaults in the FCM configuration files will most likely not be appropriate.

OIFS_GRIB_DIR - Defines the location of the GRIB library. For more details see '5.3 Install ecCodes GRIB library for OpenIFS'.

OIFS_NETCDF_DIR - Defines the location of the directory containing the netCDF libraries. This varies from system to system but typically /usr/local/lib. n.b. OpenIFS versions before 43r3 did not use the netCDF library.

OIFS_LAPACK_LIB - Defines the location of the directory containing the liblapack.a and libblas.a files. This varies from system to system but typically /usr/local/lib.

If you do not have LAPACK installed on your system or find it does not work, OpenIFS comes with all the necessary LAPACK code to run. It is disabled by default, see the README.lapack file in the OpenIFS source, or contact openifs-support@ecmwf.int for instructions on how to enable it.

Or if you prefer you can either edit the default value in the appropriate FCM configuration file in make/fcm directory or make a copy of the supplied configuration file (.fcm) and use that.

Serial (non-MPI) execution

Although the recommended way of using OpenIFS is with MPI parallel execution, we aim to provide a compilation option without MPI. This significantly affects performance and is not recommended except for specialized applications or testing. Only the lower resolutions will be realistic with a serial execution.

In a serial (non-MPI) case, the normal compiler wrappers (e.g. mpif90) are not used and the compiler (e.g. ifort) is called directly. A set of dummy subroutines is used to satisfy the linking of the model for the MPI calls.

To enable serial mode, set the following environment variable or edit oifs-config.sh directly. This enables the dummy MPI library and reset the compiler names (see READMEs/README_mpi for more details):

export OIFS_MPI=serial

Compiling OpenIFS

Once the environment variables are set, OpenIFS can be compiled (make sure you have edited your PATH environment variable to add the FCM installation bin directory). Assuming you are in the directory where the OpenIFS source code was unpacked:

% cd make
% fcm make -f oifs.fcm

The command 'fcm make' starts the compilation. The -f option specifies the location of the master configuration file for OpenIFS.

For more information about fcm command options, try: fcm --help .

This generates output similar to:

[init] make                # 2012-12-20 19:59:59Z
[init] make config-parse   # 2012-12-20 19:59:59Z
[info] config-file=...../make/oifs.fcm
[info] config-file= - ...../make/fcm/gnu-opt.fcm
[info] ............
[done] make config-parse   # 0.3s
[init] make dest-init      # 2012-12-20 19:59:59Z
[info] mode=new
[done] make dest-init      # 0.3s
[init] make build          # 2012-12-20 20:00:00Z
[info] sources: total=2194, analysed=2194, elapsed-time=11.7s, total-time=32.2s
[info] target-tree-analysis: elapsed-time=9.8s
[info] compile   targets: modified=2073, unchanged=0, total-time=1095.5s
[info] compile+  targets: modified=628, unchanged=0, total-time=1.6s
[info] ext-iface targets: modified=1247, unchanged=0, total-time=21.9s
[info] install   targets: modified=120, unchanged=0, total-time=0.3s
[info] link      targets: modified=1, unchanged=0, total-time=18.3s
[info] TOTAL     targets: modified=4069, unchanged=0, elapsed-time=316.3s
[done] make build          # 329.1s
[done] make                # 329.8s

Parallel compilation

If you have a multicore processor available, the -j <nthreads> option to 'fcm make' can be used to significantly speed up the compilation of the code.

% fcm make -j 4 -f oifs.fcm	

Tests show that compilation time reduces with a diminishing amount as the number of threads is increased. Set the number of threads to a value less than the available processor cores on your system.

Verbose compilation

The default output of FCM is quite terse. To get more verbose output use the '-v' option.  e.g.

fcm make -v -j 4 -f oifs.fcm

will produce a more detailed report and use 4 processes to analyse and build the model code in parallel.

Successful compilation

After a successful compilation, the OpenIFS executable, master.exe, can be found in the 'make/gnu-opt/oifs/bin' subdirectory (or whichever compiler you choose):

% ls
fcm  gnu-opt
% ls gnu-opt/oifs
bin   include   o
% ls gnu-opt/oifs/bin
master.exe

The 'gnu-opt' subdirectory is generated by FCM using the values of the OIFS_COMP & OIFS_BUILD environment variables. The oifs/include directory contains all the files referenced in the code by a '#include' statement. This includes the Fortran interface block files which are auto-generated by FCM and end in .intfb.h. The oifs/o directory contains all the compiled object files. The OpenIFS build does not place any additional files in the src directory (known as building 'out of source').

If using the intel compiler then the directory would be 'make/intel-opt' and so on.

Compilation problems

If you get the message:

fcm make -f oifs.fcm
/bin/ksh: fcm: not found [No such file or directory]

you need to add the 'bin' directory of the FCM installation to your PATH environment variable. FCM is supplied with OpenIFS and is in the directory 'fcm'.

We recommend editing and using the oifs-config.sh file to store all the correct settings of the OpenIFS environment variables.