Versions Compared

Old Version 57

changes.mady.by.user Unknown User (nagc)

Saved on

compared with

New Version Current

changes.mady.by.user Marcus Koehler

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...


 

Section


Please note that grib-api is being phased out in preference for ecCodes. All the grib_* commands shown below are still available with ecCodes.

grib_to_netcdf

This command will convert one or more GRIB files to netCDF and is available with the ECMWF ecCodes /grib-api software (for grib-api, versions above 1. 11.0 are recommended). For more details please see: grib_to_netcdf description.

grib_to_netcdf only works correctly when a single level coordinate is present in the GRIB file. Often the model output files have fields on multiple level types (ie. hybrid model levels and pressure levels).

Column


What are GRIB files

Excerpt

This article show different approaches to convert GRIB to netCDF for use in other analysis and visualisation software.

It also describes different methods to interpolate the reduced Gaussian grid to regular grids.

What are GRIB files

GRIB files are the file format used at ECMWF.  GRIB is a WMO standard and consists of GRIB Edition 1 and Edition 2.

The OpenIFS/IFS models output in GRIB format. These files are a mix of GRIB-1 & GRIB-2 messages, the multi-level fields are encoded as GRIB-2, whereas surface fields are GRIB-1. GRIB files can also contain multiple vertical coordinates: pressure levels, model levels, sub-surface levels etc. This can cause a problem with some 3rd party tools, as the same GRIB variable code is used for each axis. The instructions below show how to work around this by splitting the file to separate fields on different vertical axes.

OpenIFS model output

The OpenIFS model outputs two types of files: those beginning with ICMSH contain fields represented as spherical harmonics, those that begin ICMGG contain gridpoint fields.  The ICMSH files are the spherical harmonics of the wind fields, pressure and temperature and require a spectral transform to convert to gridded data. Also see controlling the OpenIFS output for more details of output fields and options.

Retrieving data from MARS archive

Please note that if using the MARS archive (apps.ecmwf.int) (e.g. for reanalysis products such as ERA-Interim/ERA-5), it is possible to download the files in netCDF format as well as GRIB.

ecCodes

/grib-api

tools

Info
Code Block
titleHow to split GRIB file into separate level types
grib_copy ICMGGftkm+001440 ICMGG_[typeOfLevel].grb

In this example, the GRIB model output file ICMGGftkm+001440 contains a number of different model level types. The special square bracket "[ ]" syntax is recognised by grib_copy (and other grib commands such as grib_filter) and can contain any valid GRIB key.

This example will copy the original file, separating the level types into their own file: ICMGG_hybrid.grb, ICMGG_isobaricInhPA.grb, ICMGG_surface ...and so on.

grib_to_netcdf can then be used on the individual files:

Code Block
titleConvert GRIB file to netCDF
grib_to_netcdf ICMGG_hybrid.grb -o ICMGG_hybrid.nc

By default, grib_to_netcdf will pack the data into scaled integers with an offset to optimize space.

If you prefer data stored as floats then use:

Code Block
languagebash
titleConvert GRIB to netCDF storing values as floats rather than scaled integers...
grib_to_netcdf -D NC_FLOAT ICMGG_hybrid.grb -o ICMGG_hybrid.nc

To convert from vorticity and divergence to wind u and v, please see CDO instructions below.

Note that grib_to_netcdf does not do any regridding. If the fields use a reduced Gaussian latitude grid, they will not be converted to a regular grid. Use CDO instead to do this as described below.


Column
width350px


Panel
borderColorgrey
bgColorwhite
titleBGColorlightlightgrey
titleOn this page

Table of Contents
indent10px


Info
The MetView Metview application is available for analysis and visualisation of OpenIFS output and will both convert spectral to gridded parameters and plot reduced Gaussian grid directly.
Please see 'Using Metview with OpenIFS' for more details.


Panel
borderColorgrey
titleSee also...

 ecCodes - grib tools

MARS  MARS user documentation

 EMOSLIB interpolation library



NCAR command language (NCL)

...

CDO supports GRIB-2 but needs either the grib_api or ecCodes library from ECMWF to be included. This is necessary in order to work correctly with OpenIFS model output.

Code Block
languagebash
titleCheck CDO is compiled with grib-api (or ecCodes)
cdo --version
Climate Data Operators version 1.79.26 (http://mpimet.mpg.de/cdo)
...........
     CDI library version Libraries: HDF5/1.710.4 proj/5.2 of Aug 10 2016 09:22:26
 CGRIBEX library version : 1.7.5 of Jun  3 2016 14:44:00
GRIB_APIxml2/2.9.4
Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5 
     CDI library version : 1.159.06
 cgribex NetCDF library version : 41.49.1 of Aug  3 2016 11:10:49 $
.........2
 ecCodes library version : 2.12.5
  NetCDF library version : 4.6.2 of Feb 14 2019 10:13:27 $
    hdf5 library version : 1.10.4 threadsafe
    exse library version : 1.4.1
    FILE library version : 1.8.3
.........

If you find the CDO commands below do not work, you can either build CDO yourself, making sure a recent version of grib_api or ecCodes is used, or you can use the workaround below.

...

Info

If you have files with a mix of GRIB-1 and GRIB-2, then either split the file first or compile cdo with "--disable-cgribex --with-grib_apieccodes=yes"

If variable names are lost, add the  "-t ecmwf" option.

...

Another way to split the file would be using the grib_copy command from the ecCodes (or grib_api) software as shown above:

Code Block
grib_copy ICMSHg4a4+000000.grb ICMSHg4a4+000000_[typeOfLevel].grb

...

Both produce a regular Gaussian grid with equal number of longitudes on each latitude row. Remember that a Gaussian grid has irregularly spaced latitudes (but nearly regular). For more details please see the description of these operators on the CDO homepage.

Convert the resulting output files to netcdf

Code Block
cdo -f nc -sp2gpl <input spectral coefficent grib>  <output gridded netcdf>

Convert the resulting output files to netcdf

After any operation converting a GRIB file to any other GRIB file, the conversion to netCDF is just:

Code Block
title
Code Block
titleConvert GRIB file to netCDF file
cdo -f nc copy <input grib> <output netcdf>

Convert vorticity and divergence to wind

To convert If you prefer wind components u & v instead of vorticity and divergence, use the cdo command:

Code Block
titleConvert vorticity & divergence to u & v
cdo dv2uvl <input file> <output file>

Interpolate to pressure from model levels

To interpolate to pressure levels from model levels, use this cdo command, for example:

Code Block
titleInterpolate model levels to pressure levels
cdo ml2pl,92500,85000,50000,20000 <input file> <output file>

...

There are various commands for inspecting the contents of a GRIB file. The GRIB_API installation as part of ecCodes software installation required for OpenIFS has useful commands like grib_ls and grib_dump.

...

Note

cdo commands can be combined into a single command for greater efficiency.

 

Possible problems and solutions

 

...


Using EMOSLIB to interpolate to regular grid

CDO's -R option, to convert from reduced Gaussian grid to regular Gaussian grid, only works with GRIB 1 format (cgribex only) (see cdo man page). This is a problem as multi-level output from OpenIFS is encoded as GRIB-2 data.

A workaround is to temporarily change the edition number to 1. Use the grib_set command (from ecCodes or grib_api) to change the GRIB edition number for all messages:

Code Block
titleWorkaround to convert GRIB 2 to regular grid
grib_set -s editionNumber=1  in.grb  in_1.grb
cdo -R copy  in_1.grb  in_1_R.grb
grib_set -s editionNumber=2  in_1_R.grb  in_2_R.grb

This does not actually convert the message to GRIB-1, it only changes the editionNumber. Any GRIB-2 specific elements are unaltered.

It's recommended that the editionNumber is changed back to 2 to ensure that the parameter names remain correct in any subsequent cdo / grib commands.

Workaround if using more than 128 levels

The workaround above will fail if the grib file contains field with more than 128 levels. This is because there are not enough bits available in the GRIB-1 parameter to encode the total number of model half levels (which would exceed 255).

In this case, the 'pv' GRIB parameter causes the problem (nothing to do with potential vorticity!). This is an array holding the half level values of the model's A & B coefficients that define the location of the levels.

The workaround above can be extended to delete this array while setting the edition number to 1, like this:

The ECMWF interpolation software library EMOSLIB also provides the capability to interpolate spectral data to regular gaussian grids or regular lat-lon grids, and interpolate regular gaussian grids to regular lat-lon grids.

The EMOSLIB library provides a Fortran library to enable users to write their own interpolation software. Please see the EMOSLIB website for more details and examples.

EMOSLIB also provides two command-line tools for using interpolation. Assuming a recent version of the EMOS library, there are two tools that can be used.

In the 'bin' directory there is a command 'emos_tool' which can be used as:

Code Block
titlebin/emos_tool example
emos_tool --regular=256 [--area="40/-10/31/20"]  in.grib out.grib

The grid specified for the --regular option follows the grid naming convention for EMOSLIB here: Reduced Gaussian Grids.

Another command can be found in the 'tools' directory. This allows the interpolation function to be specified and is more flexible:

Code Block
tools/int --INTOUT:gridname=F256 --INTOUT:area="40/-10/31/20" --input=1.grib --output=2.grib --intf2

Here 'F256' means 'full-grid' not reduced, and --intf2 is the EMOSLIB interpolation function to be used.

Possible problems and solutions


cdo -R option does not work with GRIB-2 fields to convert to regular grid

CDO's -R option, to convert from reduced Gaussian grid to regular Gaussian grid, only works with GRIB 1 format as the CGRIBEX decoder this uses does not work with GRIB-2 (see cdo man page). This is a problem as multi-level output from OpenIFS is encoded as GRIB-2 data.

The preferred way to deal with this is to use the setgridtype operator as described above:

Code Block
cdo -f nc setgridtype,regular <input grib> <output netcdf>

However, if this does not work for any reason, a workaround is to temporarily change the edition number to 1 of the GRIB file. Use the grib_set command (from ecCodes) to change the GRIB edition number for all messages:

Code Block
titleWorkaround to convert GRIB 2 to regular grid
grib_set -s editionNumber=1  in.grb  in_1.grb
cdo -
Code Block
grib_set -s deletePV=1,editionNumber=1  in.grb  in_1.grb
cdo -R copy  in_1.grb  in_1_R.grb
grib_set -s editionNumber=2  in_1_R.grb  in_2_R.grb

The deletePV option is known as a 'concept' rather than a parameter contained in the GRIB file itself. It ensures the PV array is deleted correctly by grib_set. Now the cdo regridding will work correctly, but remember to reset the editionNumber back to 2.

The A & B half level coefficients can be recovered from the original file by:

Code Block
cdo vct in.grb > pv.txt

 

Parameter names are lost for GRIB-1 fields

ECMWF GRIB-1 use keys which may not be recognised by CDO because they are locally defined (e.g. the shortName key) and not defined in the WMO GRIB tables that CDO uses.  This can cause parameter names to be lost or not recognised when using the CDO commands.

A workaround is to use the -t ecmwf option.  The -t option tells CDO to use the predefined ECMWF parameter tables (see CDO documentation for more details).

Code Block
cdo -t ecmwf -f nc copy mygrib1.grb mygrib1.nc
Warning

Do not use this option for GRIB-2 fields. It sets the GRIB table default to be specific to the GRIB-1 ECMWF tables. If problems persist, we recommend using grib_to_netcdf to convert to netCDF.

 

 

Acknowledgements

Thanks to Paul Dando of User Support for help with the contents of this page.

 

 

...

This does not actually convert the message to GRIB-1, it only changes the editionNumber. Any GRIB-2 specific elements are unaltered.

It's recommended that the editionNumber is changed back to 2 to ensure that the parameter names remain correct in any subsequent cdo / grib commands.

Workaround if using more than 128 levels

The workaround above will fail if the grib file contains field with more than 128 levels. This is because there are not enough bits available in the GRIB-1 parameter to encode the total number of model half levels (which would exceed 255).

In this case, the 'pv' GRIB parameter causes the problem (nothing to do with potential vorticity!). This is an array holding the half level values of the model's A & B coefficients that define the location of the levels.

The workaround above can be extended to delete this array while setting the edition number to 1, like this:

Code Block
grib_set -s deletePV=1,editionNumber=1  in.grb  in_1.grb
cdo -R copy  in_1.grb  in_1_R.grb
grib_set -s editionNumber=2  in_1_R.grb  in_2_R.grb

The deletePV option is known as a 'concept' rather than a parameter contained in the GRIB file itself. It ensures the PV array is deleted correctly by grib_set. Now the cdo regridding will work correctly, but remember to reset the editionNumber back to 2.

The A & B half level coefficients can be recovered from the original file by:

Code Block
cdo vct in.grb > pv.txt


Parameter names are lost for GRIB-1 fields

ECMWF GRIB-1 use keys which may not be recognised by CDO because they are locally defined (e.g. the shortName key) and not defined in the WMO GRIB tables that CDO uses.  This can cause parameter names to be lost or not recognised when using the CDO commands.

A workaround is to use the -t ecmwf option.  This will make CDO use ECMWF parameter table 128 for the definitions of variables but 128 is only for GRIB 1. This will work for the surface fields which are encoded by IFS as GRIB 1, but not for the multi-level fields which are encoded in GRIB 2. Variable names become 'unknown' if -t ecmwf is used with the multi-level fields. It is best to omit it completely, even for surface fields, and only use it for GRIB 1 data if you get missing variable names after CDO has converted the data (see CDO documentation for more details).

Code Block
cdo -t ecmwf -f nc copy mygrib1.grb mygrib1.nc


Warning

Do not use this option for GRIB-2 fields. It sets the GRIB table default to be specific to the GRIB-1 ECMWF tables. If problems persist, we recommend using grib_to_netcdf to convert to netCDF.


Acknowledgements

Thanks to Paul Dando of User Support for help with the contents of this page.




HTML Comment
hiddentrue

The excerpt below includes a Google analytics tracker, do not remove!

Excerpt Include
Credits
Credits
nopaneltrue

...