It also describes different methods to interpolate the reduced Gaussian grid to regular grids.
What are GRIB files
GRIB files are the file format used at ECMWF. GRIB is a WMO standard and consists of GRIB Edition 1 and Edition 2.
The OpenIFS/IFS models output in GRIB format. These files are a mix of GRIB-1 & GRIB-2 messages, the multi-level fields are encoded as GRIB-2, whereas surface fields are GRIB-1. GRIB files can also contain multiple vertical coordinates: pressure levels, model levels, sub-surface levels etc. This can cause a problem with some 3rd party tools, as the same GRIB variable code is used for each axis. The instructions below show how to work around this by splitting the file to separate fields on different vertical axes.
OpenIFS model output
The OpenIFS model outputs two types of files: those beginning with ICMSH contain fields represented as spherical harmonics, those that begin ICMGG contain gridpoint fields. The ICMSH files are the spherical harmonics of the wind fields, pressure and temperature and require a spectral transform to convert to gridded data. Also see controlling the OpenIFS output for more details of output fields and options.
Retrieving data from MARS archive
Please note that if using the MARS archive (apps.ecmwf.int) (e.g. for reanalysis products such as ERA-Interim/ERA-5), it is possible to download the files in netCDF format as well as GRIB.
This command will convert one or more GRIB files to netCDF and is available with the ECMWF ecCodes software. For more details please see: grib_to_netcdf description.
grib_to_netcdf only works correctly when a single level coordinate is present in the GRIB file. Often the model output files have fields on multiple level types (ie. hybrid model levels and pressure levels).
In this example, the GRIB model output file
ICMGGftkm+001440 contains a number of different model level types. The special square bracket "[ ]" syntax is recognised by
grib_copy (and other grib commands such as
grib_filter) and can contain any valid GRIB key.
This example will copy the original file, separating the level types into their own file:
ICMGG_surface ...and so on.
grib_to_netcdf can then be used on the individual files:
By default, grib_to_netcdf will pack the data into scaled integers with an offset to optimize space.
If you prefer data stored as floats then use:
To convert from vorticity and divergence to wind u and v, please see CDO instructions below.
grib_to_netcdf does not do any regridding. If the fields use a reduced Gaussian latitude grid, they will not be converted to a regular grid. Use CDO instead to do this as described below.
Please see 'Using Metview with OpenIFS' for more details.
NCAR command language (NCL)
NCL provides a tool to convert both GRIB-1 & GRIB-2 to netCDF called ncl_convert2nc.
NCL example scripts to convert to netCDF are also available.
CDO: Climate Data Operators
These instructions assume the use of a recent version of Climate Data Operators (CDO) (available from Max-Planck-Institut, Germany).
CDO supports GRIB-2 but needs either the ecCodes library from ECMWF to be included. This is necessary in order to work correctly with OpenIFS model output.
If you find the CDO commands below do not work, you can either build CDO yourself, making sure a recent version of ecCodes is used, or you can use the workaround below.
Interpolate from reduced to regular grid
CDO can be used to interpolate from the reduced Gaussian grid to a regular Gaussian grid:
-R option only works for GRIB edition 1 (GRIB 1) data as it uses the CGRIBEX decoder.
For GRIB-2 messages it's better to use the
If you have files with a mix of GRIB-1 and GRIB-2, then either split the file first or compile cdo with "--disable-cgribex --with-eccodes=yes"
If variable names are lost, add the "-t ecmwf" option.
If these steps do not work, see workarounds below.
Steps to convert GRIB to netCDF
Split z axis
Before converting to netCDF, separate the different vertical axes using the generic command:
cdo splitzaxis <input file> <output file pattern>.
This will ensure the following steps work correctly:
In this example, the different files might contain: '01' - fields on pressure levels, '02' - fields on model levels, '03' - hybrid levels, '04' - surface and so on. The number of files created depends on the number of different types of levels. Use the command
grib_ls to inspect the contents of each split file.
Note that the filename
ICMSH in this example, indicates it contains spectral and not gridpoint fields.
Another way to split the file would be using the grib_copy command from the ecCodes software as shown above:
In this case 'typeOfLevel' is a GRIB key. The square brackets is a special syntax to the
grib_copy command. This approach works with any GRIB key.
Convert spectral to gridpoint
CDO supports two options for converting spectral to gridpoint data:
- sp2gpl - converts spectral to a linear Gaussian grid, appropriate for IFS data.
- sp2gp - converts spectral to a quadratic Gaussian grid (only use this for T21/42 resolutions)
Both produce a regular Gaussian grid with equal number of longitudes on each latitude row. Remember that a Gaussian grid has irregularly spaced latitudes (but nearly regular). For more details please see the description of these operators on the CDO homepage.
Convert the resulting output files to netcdf
After any operation converting a GRIB file to any other GRIB file, the conversion to netCDF is just:
Convert vorticity and divergence to wind
To convert wind components u & v instead of vorticity and divergence, use:
Interpolate to pressure from model levels
To interpolate to pressure levels from model levels, use:
Viewing contents of GRIB files
There are various commands for inspecting the contents of a GRIB file. The ecCodes software installation required for OpenIFS has useful commands like
Here is another example using the cdo command:
Converting a single file
In this example a single GRIB file with model level data, on a regular Gaussian grid is converted to netCDF with vorticity & divergence replaced by u & v and model level data interpolated to pressure levels.
Converting series of files
The following script shows how to loop over several output files using the steps above to convert to netCDF. Note the spectral parameter files are also converted but the extra command using the sp2gpl operator is first used to convert the variables from spectral to gridpoint on a regular, linear, Gaussian grid.
In this example, we use the pipelining feature of CDO to do both the 'copy' and 'sp2gpl' steps in one command, keeping the entire operations in memory for efficiency.
cdo commands can be combined into a single command for greater efficiency.
Using EMOSLIB to interpolate to regular grid
The ECMWF interpolation software library EMOSLIB also provides the capability to interpolate spectral data to regular gaussian grids or regular lat-lon grids, and interpolate regular gaussian grids to regular lat-lon grids.
The EMOSLIB library provides a Fortran library to enable users to write their own interpolation software. Please see the EMOSLIB website for more details and examples.
EMOSLIB also provides two command-line tools for using interpolation. Assuming a recent version of the EMOS library, there are two tools that can be used.
'bin' directory there is a command
'emos_tool' which can be used as:
The grid specified for the
--regular option follows the grid naming convention for EMOSLIB here: Reduced Gaussian Grids.
Another command can be found in the
'tools' directory. This allows the interpolation function to be specified and is more flexible:
Here 'F256' means 'full-grid' not reduced, and
--intf2 is the EMOSLIB interpolation function to be used.
Possible problems and solutions
cdo -R option does not work with GRIB-2 fields to convert to regular grid
CDO's -R option, to convert from reduced Gaussian grid to regular Gaussian grid, only works with GRIB 1 format as the CGRIBEX decoder this uses does not work with GRIB-2 (see cdo man page). This is a problem as multi-level output from OpenIFS is encoded as GRIB-2 data.
The preferred way to deal with this is to use the setgridtype operator as described above:
However, if this does not work for any reason, a workaround is to temporarily change the edition number to 1 of the GRIB file. Use the
grib_set command (from ecCodes) to change the GRIB edition number for all messages:
This does not actually convert the message to GRIB-1, it only changes the editionNumber. Any GRIB-2 specific elements are unaltered.
It's recommended that the
editionNumber is changed back to 2 to ensure that the parameter names remain correct in any subsequent cdo / grib commands.
Workaround if using more than 128 levels
The workaround above will fail if the grib file contains field with more than 128 levels. This is because there are not enough bits available in the GRIB-1 parameter to encode the total number of model half levels (which would exceed 255).
In this case, the 'pv' GRIB parameter causes the problem (nothing to do with potential vorticity!). This is an array holding the half level values of the model's A & B coefficients that define the location of the levels.
The workaround above can be extended to delete this array while setting the edition number to 1, like this:
The deletePV option is known as a 'concept' rather than a parameter contained in the GRIB file itself. It ensures the PV array is deleted correctly by grib_set. Now the cdo regridding will work correctly, but remember to reset the editionNumber back to 2.
The A & B half level coefficients can be recovered from the original file by:
Parameter names are lost for GRIB-1 fields
ECMWF GRIB-1 use keys which may not be recognised by CDO because they are locally defined (e.g. the
shortName key) and not defined in the WMO GRIB tables that CDO uses. This can cause parameter names to be lost or not recognised when using the CDO commands.
A workaround is to use the
-t ecmwf option. This will make CDO use ECMWF parameter table 128 for the definitions of variables but 128 is only for GRIB 1. This will work for the surface fields which are encoded by IFS as GRIB 1, but not for the multi-level fields which are encoded in GRIB 2. Variable names become 'unknown' if -t ecmwf is used with the multi-level fields. It is best to omit it completely, even for surface fields, and only use it for GRIB 1 data if you get missing variable names after CDO has converted the data (see CDO documentation for more details).
Do not use this option for GRIB-2 fields. It sets the GRIB table default to be specific to the GRIB-1 ECMWF tables. If problems persist, we recommend using grib_to_netcdf to convert to netCDF.
Thanks to Paul Dando of User Support for help with the contents of this page.