- Created by Unknown User (nagc), last modified on Sep 18, 2020
Introduction
This article describes how the compilation of the OpenIFS model is controlled and how to modify the compilation. It provides more detailed information than the page 5.5 Building OpenIFS. There are a number of ways in which the compilation can be changed.
Various compiler and configuration files are supported for OpenIFS which are tested and known to work on systems we have access to (see 5.1 Supported systems).
The supplied configurations can be edited or over-ridden as described below, but we caution the user not to try their 'favourite' options. IFS is a complex code and may not behave appropriately with different compile options than supplied.
Compiling with FCM
The Flexible Configuration Management (FCM) software, from the UK Meteorological Office, is used to compile the model. FCM offers a number of advantages: configuration files are straightforward to understand, model object files are compiled 'out-of-source', source code changes can be 'out-of-source', fortran interface blocks can be generated automatically.
FCM replaces the traditional 'make' or 'gmake' command and is specifically written for compiling large fortran or mixed fortran/C codes like OpenIFS.
For more information about FCM, please see the FCM User Guide (from the UK Met Office).
Configuration files and variables
The FCM software uses configuration files identified by the suffix '.fcm' (n.b. OpenIFS versions before 43r3 used .cfg as the suffix). They can be found in the directory oifs/make/fcm
.
The master configuration file for OpenIFS is in make/oifs.fcm
. It sets general configuration options for FCM, lets FCM know where the source code is and reads a single architecture and compiler specific configuration file. No compiler options are contained in the oifs.fcm
file - these are always in the architecture & build specific configuration files.
The naming convention for these architecture specific files is: <compiler>-<build type>.fcm. For example, the file fcm/gnu-opt.fcm
would be for the GNU compilers and the optimized build. The user can copy and rename the files provided or create their own using an existing one as a template.
Build types
The supplied FCM configuration files support 3 different build types:
Optimized build ('opt') - this is uses higher optimization and will generate the faster running model. Note that we do not always recommend the very highest optimization levels as this can cause the model to fail in select subroutines. The optimizations options are chosen conservatively to ensure the model will run successfully on a wide range of hardware and compiler versions.
OpenMP is enabled in this configuration only.
Non-optimized build ('noopt') - this is intended for testing, development and debugging. Optimization is set to -O0 and OpenMP is disabled. The model will run significantly slower. We recommend if the model fails with OIFS_BUILD=opt when compiling OpenIFS for the first time, this build configuration should be used as it will compile faster and be more stable.
Full debugging ('nansC') - this configuration is intended for debugging only. As well as setting the lowest optimization, -O0, it also enables array bound checking and initialisation of variables with special values that will trigger 'not-a-number' exceptions useful for trapping variables used before initialized. This will cause the model to run very slowly and is normally only used for short runs for tracing bugs. This configuration is not guaranteed to work with all model versions.
How to change build type.
Before compiling the OpenIFS model, the build configuration must first be set by several environment variables:
OIFS_COMP - This sets the choice of compiler. The default is set in make/fcm configuration files to 'gnu' which means the gfortran/gcc compilers will be used.
OIFS_BUILD - This sets the type of build. The default is 'noopt'.
export OIFS_COMP=intel export OIFS_BUILD=opt
These variables and their defaults are set in the make/oifs.fcm
file. e.g.
$OIFS_COMP{?} = gnu $OIFS_BUILD{?} = noopt
Overriding defaults in configuration files
The FCM syntax allows for variables in the configuration files (.fcm) to be overridden by user environment variables. You will have already seen this before when setting OIFS_BUILD
and OIFS_COMP
which correspond to these lines in the make/fcm/*.fcm
files.
The {?}
syntax (often seen in the configuration files) means that if an environment variable of the same name exists, that value will be used instead of the value assigned. This provides a way to change many of the settings in the configuration files.
Create your own config files
Suppose we wanted to run the model on MacOS X with the intel compiler and a optimized setting. There are several ways we could change the compilation configuration.
(a) We could set:
export OIFS_COMP=osx_intel export OIFS_BUILD=opt
FCM would then expect to find a file called: osx_intel-opt.cfg in the make/cfg directory
. This file would need to be created by the user. We suggest copying one of the existing .cfg files as a template and modifying it to suit your purpose.
(b) Or you could use the OIFS_CFG
environment variable. By default this variable is set to:
$OIFS_CFG{?} = $HERE/fcm/${OIFS_COMP}-${OIFS_BUILD}.fcm
The $HERE
variable in this line means 'the directory where this file is located'.
Instead of changing OIFS_COMP
& OIFS_BUILD
, you could instead use:
export OIFS_CFG="\$HERE/fcm/mac-gnu-opt.fcm
The mac-gnu-opt.fcm
would need to be an edited copy of one of the configuration files in the make/fcm
directory. The backslash character '\' means the '$' symbol is passed through to the fcm command rather than being interpreted by the export command.
Your own .fcm files do not need to be kept in the make/fcm
directory. You might like to create a directory, say, make/my-configs
and store them there. Use the -f option for the fcm make command to load the new files:
% cd oifs/make % mkdir my-configs % cp fcm/intel-opt.fcm my-configs/osx_intel_v13-opt.fcm (edit osx_intel_v13-opt.fcm) % export OIFS_CFG="\$HERE/my-configs/osx_intel_v13-opt.fcm" % fcm make -f mycfg/oifs.cfg
Note there is no need to set OIFS_BUILD
& OIFS_COMP
in this example as they are not used, OIFS_CFG
takes precedence.
Change compiler options globally
All the compiler flags and options are contained in the build specific files (e.g. intel-opt.fcm). As above, a number of environment variables can be defined to override the default options set in these files.
As a general rule, any variable with a {?} can be overriden by setting an environment variable.
The following compilation variables in the configuration files can be overriden by environment variables (this may vary between releases so always check the .fcm file):
User configurable variables | Description |
---|---|
OIFS_GRIB_DIR | Location of the GRIB library installation directory. This must be changed or overridden to the location of your ecCodes install before OpenIFS can be compiled. |
OIFS_GRIB_INCLUDE | Location of the 'include' sub-directory in the eccodes installation. This does not normally need changing unless it's located in a non-standard location. |
OIFS_GRIB_LIB | Location of the 'lib' sub-directory in the eccodes installation. This does not normally need changing. Note: although eccodes builds both shared and static libraries, OpenIFS always links to the static library in case different versions of eccodes exist on the user system. |
OIFS_NETCDF_DIR | Location of the netCDF installation directory. This must be changed or overridden to the location on your system as the default supplied may not be correct. |
OIFS_LAPACK_LIB | Compiler options to set the directory of the LAPACK and BLAS libraries. This often needs to be changed or overriden as the supplied default is unlikely to be correct on most systems. |
OIFS_EXTRA_LIB | Available in case additional libraries need to be specified. This is most likely set when vendors supply special optimized libraries. One example is IBM's optimized math function library MASS. e.g. OIFS_EXTRA_LIB{?} = -L/usr/lib/mass -lmass -lmassvp6 (n.b. only OpenIFS version before 43r3 supported the IBM compiler). |
OIFS_FC | Specifies the fortran compiler. The normal default is 'mpif90' but might be 'mpiifort' for the Intel compiler or more generally 'mpifc'. We recommend that mpi compiler wrappers are used whenever possible to simplify the compilation e.g. OIFS_FC = mpif90. |
OIFS_FFLAGS | Fortran compiler flags for free format source files. These flags will vary between the 'opt', 'noopt' and 'nansC' build configurations. e.g. OIFS_FFLAGS = -g -O2 -m64 -march=native -fconvert=big-endian -fopenmp |
OIFS_FFIXED | Extra fortran compiler flags, in addition to OIFS_FFLAGS, for fixed source form Fortran. There are a small number of fortran source files in OpenIFS that are still in fixed format. These options will be added to the options in OIFS_FFLAGS. |
OIFS_FCDEFS | Fortran preprocessor DEFINE flags. This is a list, separated by spaces, of the DEFINE flags required by the code in preprocessor statements like #ifdef. The user is advised not to change these |
OIFS_LFLAGS | Linker options. These are separate flags to be passed to the linker. For example, a common requirement is to generate a loadmap or increase stack memory size. e.g. OIFS_LFLAGS{?} = -bloadmap:map -bmaxstack:0x80000000 |
OIFS_CC | Specifies the C compiler. As for the Fortran compiler this would normally be a compiler wrapper supplied by the MPI system in use. e.g. mpicc for GCC or mpiicc for Intel. |
OIFS_CFLAGS | C compiler flags. As for the Fortran flags, these will vary between the 'opt', 'noopt' and 'nansC' build options. |
OIFS_CCDEFS | C preprocessor flags. As for the OIFS_FCDEFS, the user is advised not to change these. |
Changing compiler options per file
The description above covers changing the compiler options globally, that is, for all files in the compilation. There may be instances when options for one or a few files need to be changed. For instance, in debugging where array bound checking is required for a handful of changed subroutines.
Changing options per file cannot be done using environment variables. There are several ways in which it can be done:
Edit the oifs.fcm file
The oifs.fcm
file is the place to put all file specific options. Some can be found in this file already e.g.
oifs.prop{fc.flags}[algor/internal/fourier] = $OIFS_FFIXED $OIFS_FFLAGS oifs.prop{fc.flags}[algor/internal/lanczos/i2x.F] = $OIFS_FFIXED $OIFS_FFLAGS
The first line above sets the build property 'fc.flags
' (ie. fortran compiler flags) for the directory algor/internal/fourier
(under the 'src
' directory). Note this sets compiler flags for all files in that directory.
The second line set the build property, compiler flags, for the specific file, algor/internal/lanczos/i2x.F
. Note that all other files in this directory, unless changed elsewhere, will use the default compiler options as described above.
These statements are used to ensure F77 files are compiled correctly.
Example: you need to reduce the optimisation and include array bound checking on a specific file for testing. We suggest you make a copy of the oifs.fcm
and add the line:
oifs.prop{fc.flags}[ifs/module/yomtrans.F90] = -g -O0 -m64 -C -fconvert=big-endian
and then compile using this new file:
fcm make -f oifs_copy.fcm
As this is a new .fcm file for FCM it will do a full build of the code from scratch. Any subsequent builds will be incremental - only compiling changed files.
Using the 'include' statement
FCM configuration files support an 'include
' statement. The oifs.fcm
file uses this to include architecture and compiler specific information. If you need to include several lines changing the options for multiple files then another option would be to add an additional 'include
' statement to point to a new file containing the oifs.prop
lines you need. For example:
include = $HERE/debug.fcm
and then create the file make/fcm/debug.fcm
with the 'oifs.prop
' lines you need.
The $HERE variable is a special FCM specific variable that expands to the directory name containing the file. In this example the oifs.fcm
and your debug.fcm
are in the same directory.
Inherited build
A third way to customize at the file level is through FCM's 'inherited build' mechanism. This simple but powerful facility is described in more detail in the HowTo modify and add new code. In brief this allows you to leave the current configuration and source untouched and 'use' or 'inherit' the OpenIFS configuration in a completely separate directory. This makes it easier to update the OpenIFS code for new releases and keeps any changes you make completely separate.
On this page ...