Versions Compared
Key
- This line was added.
- This line was removed.
- Formatting was changed.
Introduction
The ECMWF ecCodes software librarypackage provides a set of functions/subroutines and command line tools for working with WMO FM-92 GRIB edition 1 and edition 2 messages. The OpenIFS models use GRIB for their input and output files.
The ecCodes software library package must be built and installed before compiling any OpenIFS model that needs it.
To download the software please visit the ecCodes website.
If you have a version of ecCodes or the older (no longer supported) ECMWF grib_api library on your system that would most likely work, though check the minimum grib-api version required for each model. It is recommended that you build and install ecCodes as described below to ensure the correct configuration is used. We also recommend compiling your own to ensure the same compiler and compiler version is used for the model, as otherwise, error messages about conflicting module versions when compiling can arise.
Info |
---|
The older ECMWF grib_api library can be used with OpenIFS, but as it's no longer developed by ECMWF, ecCodes is recommended. Versions of OpenIFS before 40r1v2 may need a small code change to work with ecCodes. |
Before you start
Decide where you will put the ecCodes software.
For a single user, ecCodes might be installed in your home directory. For a multi-user environment, say a cluster or High Performance Computing Facility (HPCF), installing ecCodes centrally once only is probably best, then point your build configuration to this location.
In the walk-through example below, a single-user download and install is assumed.
Info | ||
---|---|---|
| ||
OpenIFS does not need the 'JPEG' or 'PNG' facility in ecCodes as the model does not use any image based data. They can be disabled to remove the dependency on libjpeg and libpng (see below). |
Info | ||
---|---|---|
| ||
ecCodes can be build and linked against python2. However, users are recommended to turn this option off and use the Python3 interface to eccodes that can be installed from PyPI after eccodes is installed (see below). |
Download and unpack
Create a directory called $HOME/ecmwf
in which ecCodes will be downloaded and compiled.
If you haven't already obtained ecCodes, get it from the ecCodes Releases page. We recommend downloading the latest version.
To unpack the software (2.21.0 is given as an example, please use the latest version number):
Code Block | ||||
---|---|---|---|---|
| ||||
% mkdir -p ecmwf % cd ecmwf % tar zxf eccodes-2.921.0-Source.tar.gz |
Configure ecCodes using CMake
ecCodes uses the CMake command for configuration and compilation. CMake is usually installed by default on modern Linux systems but may have to be installed if not. Note that a cmake version 3.6 and above is required. Some older linux systems may have earlier versions.
CMake compilation is done in a separate build directory to both the source and the install directory. More information on configuring and building ecCodes is available on the ecCodes page.
First create a new directory, "eccodes" in this example, go into it and then run the cmake command to install the build files into that directory. In this example we'll also install the compiled ecCodes into the same directory, but you can change it to by altering the -DCMAKE_INSTALL_PREFIX (e.g. /usr/local/eccodes).
Code Block | ||||
---|---|---|---|---|
| ||||
Code Block | ||||
| ||||
% mkdir bld % mkdir eccodes % cd bldeccodes % cmake ../eccodes-2.921.0-Source \ -DCMAKE_C_COMPILER=gcc \ -DCMAKE_Fortran_COMPILER=gfortran \ -DCMAKE_INSTALL_PREFIX="$HOME/ecmwf/eccodes" \ -DENABLE_NETCDF=ON /ecmwf/eccodes" \ -DENABLE_JPGNETCDF=OFF ON \ -DENABLE_PNGJPG=OFF \ -DENABLE_PYTHONPNG=OFF \ -DENABLE_FORTRAN=ON \ -DBUILD_SHARED_LIBS=BOTH |
Explanation of lines. Note that all options are prefixed by '-D':
| This sets the choice of compiler. CMake is often able to determine the available compilers for itself. Note the CMake variables are preferred rather than set environment variables CC and FC. | ||
CMAKE_INSTALL_PREFIX="$HOME/ecmwf/eccodes" | This specifies the location where the 'make install' command will place the eccodes installation. | ||
ENABLE_NETCDF=ON | Enabling this option ensures the grib_to_netcdf command can be used. Note: a netCDF library must be available on the system. | ||
ENABLE_JPG=OFF ENABLE_PNG=OFF =OFF | Setting these As OpenIFS models do not deal with any image based GRIB data these options can be disabled, unless you have other GRIB data containing images that you need to work with. Setting this removes the need for the Jasper, OpenJPG or PNG system libraries. | ENABLE_PYTHON=OFF | Some of the utilities that come with the OpenIFS model make use of the python interface to ecCodes e.g. the tools to create and manipulate the model data. If this option is enabled it compiles the Python2 interface which is now deprecated. Users are recommended to turn this option off and use the Python3 interface to eccodes that can be installed from PyPI after eccodes is installed system libraries, which can simplify ecCodes installation. The use of ecCodes in OpenIFS does not need these options, though other GRIB data encoded using JPEG and PNG will not be supported. |
ENABLE_FORTRAN=ON | This enables the Fortran interface to ecCodes and should always be on as OpenIFS uses this. | ||
BUILD_SHARED_LIBS=BOTH | The ecCodes library can be built with both 'shared' and 'static' libraries for linking, by default only shared libraries are built. Shared libraries may be needed for the python interface depending on your installation. Some compilers will link to the shared version by default if both shared and static are found (e.g. the GNU compiler). Disabling shared libraries from being generated is not recommended as it may prevents the ecCodes command line tools from working. Note some IBM users may experience problems if shared libraries are builtmay prevents the ecCodes command line tools from working. Note some IBM users may experience problems if shared libraries are built. |
Info |
---|
Please note that the python interface to ecCodes can be installed from PyPi for python3. For users who need a python2 interface, please include the option 'ENABLE_PYTHON2=ON'. For more information about using ecCodes with python3, please see: ecCodes installation. |
Available options
To get additional help on the available options and build configuration run the command:
Code Block |
---|
% cmake -LH eccodes-2.9.0-Source.21.0-Source |
Change the version number to your installation.
Info |
---|
This command will cause cmake to run its system discovery if this is the first time the command has been used. This may take time to complete and generate additional files. |
Preferred compiler options
By default, cmake will use the preferred compiler and set appropriate compiler options accordingly (for cmake, this depends on the build type).
This may not be what you need if you have multiple compilers available and want to direct cmake to the most appropriate one to use.
Recommended options given below are conservative to ensure stability and bit-reproducibility.
Changing compiler options is not normally necessary. However, the following examples show the recommended compiler options and how to set them for the cmake commands.
Panel | |||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||||||||||
Compiler options if using cmake
|
Compiling, checks and installing
After configuration the next steps are to compile ecCodes and install it. To compile ecCodes do:
Code Block |
---|
% make -j2 2>&1 | tee make.out |
which will send all the standard and error output from the make command to the file 'make.out' as well as the terminal. This is recommended as the output is lengthy.
The -j2 option will use 2 threads (cores) for a faster compilation.
Info | ||
---|---|---|
| ||
If using cmake, it will normally hide all the output from the compile commands. If you want to see the compile command (to check the compile options), do: make VERBOSE=1 |
Info | ||
---|---|---|
| ||
If your computer has multiple cores, use the -j flag to make to build ecCodes faster. your computer has multiple processors (cores), we recommend using more cores for faster compilation (specifying more cores than the computer has will slow the compilation down) e.g.
|
Next, to verify ecCodes works correctly run (this will likely download some test data from the ECMWF test-data server):
Code Block |
---|
% makectest check-j2 | tee checkctest.out |
Verify that all the ecCodes tests have passed. If not, go back and check your configuration options. In case of difficulty, contact Software.Support@ecmwf.int.
Finally, to install ecCodes do (the installation directory is specified in the -DCMAKE_INSTALL_PREFIX option (see above):
Code Block |
---|
% make install | tee install.out |
At the end of this step, in the directory $HOME/ecmwf/
that we use in this example, you should now have a directory called eccodes which contains the following:
Code Block |
---|
% ls eccodes bin include lib share % ls eccodes/lib libeccodes.a libeccodes_f90.a pkgconfig python2.7 |
Verify the installation
To verify that the installation was successful, first ensure that the eccodes bin directory is added to your PATH environment variable and then run the codes_info command e.g.
Code Block |
---|
% export PATH=$PATH:$HOME/ecmwf/eccodes/bin % codes_info ecCodes Version 2.921.0 Default definition files path is used: /home/myname/ecmwf/eccodes/share/eccodes/definitions Definition files path can be changed setting ECCODES_DEFINITION_PATH environment variable Default SAMPLES path is used: /home/myname/ecmwf/eccodes/share/eccodes/samples SAMPLES path can be changed setting ECCODES_SAMPLES_PATH environment variable |
Your output may differ for different ecCodes versions. If the installation was successful, codes_info will report the installation directories.
Possible problems
Info | ||
---|---|---|
| ||
OpenIFS support: openifs-support@ecmwf.int. |
Cray
Possible issues with the Cray systems (using cce or gnu compilers) are related to building shared libraries and linking with supporting netcdf and python libraries.
1. If problems occur, first try using the option "-DBUILD_SHARED_LIBS=OFF" to only build statically linked libraries.
2. If linking problems still occur, try unloading the Cray ATP module which has been known to cause problems.
Code Block |
---|
module unload atp |
3. -llibm link error.
If the following occurs:
Code Block | ||
---|---|---|
| ||
/usr/bin/ld: attempted static link of dynamic object `/usr/lib64/libm.so' |
set the environment variable CRAYPE_LINK_TYPE=dynamic (or add the compiler option '-dynamic').
4. Missing python library.
Although it's recommended not to compile with python enabled, if you need this and If this error occurs when compiling _gribapi_swig.so:
Code Block |
---|
/usr/bin/ld: cannot find -lpython |
it means the directory path of the python library is not on the linkers list of directories to search.
Make sure the python module is loaded. Then check any PYTHON environment variables set by the module and confirm the location of the libpython (or other version) library.
Set the environment variable LIBRARY_PATH to the correct directory so that the linker can find it:
Code Block | ||
---|---|---|
| ||
export LIBRARY_PATH=/home/anaconda/2.2.0-python2/lib64 |
LIBRARY_PATH
is only used at compile/link time. It may also be necessary to add this directory to the LD_LIBRARY_PATH
environment variable which is searched for libraries at runtime.
Building on High Performance Computer Systems
Some HPC batch systems have a different hardware architecture for their login (or frontend node) to the batch node, but the frontend compilation system is targeted at the batch nodes. This is known as cross-compilation. If this is the case you may see failures in the 'make check' stage because the checks, although compiled for the backend batch nodes, are being run on the frontend nodes and therefore may not work correctly. If this is the case on your system, we recommend using a batch job to do the 'cmake; make; make check; make install' steps.
Note if you plan on using the eccodes software in your own software that runs on the frontend nodes you may need to compile and install eccodes twice; one for the batch system and again for the frontend system.
If you have any questions installing eccodes in this type of environment, please contact: openifs-support@ecmwf.int for assistance.
Excerpt Include | ||||||
---|---|---|---|---|---|---|
|
Panel | ||||
---|---|---|---|---|
| ||||
On this page ...
|
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||
|
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||
|