...
Section | |||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
NCAR command language (NCL)
...
CDO supports GRIB-2 but needs either the grib_api or ecCodes library from ECMWF to be included. This is necessary in order to work correctly with OpenIFS model output.
Code Block | ||||
---|---|---|---|---|
| ||||
cdo --version Climate Data Operators version 1.79.26 (http://mpimet.mpg.de/cdo) ........... CDI library version Libraries: HDF5/1.7.2 of Aug 10 2016 09:22:26 CGRIBEX library version : 1.7.5 of Jun 3 2016 14:44:00 GRIB_API10.4 proj/5.2 xml2/2.9.4 Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5 CDI library version : 1.159.06 cgribex library version : 1.9.2 ecCodes library version : 2.12.5 NetCDF library version : 4.46.12 of AugFeb 14 32019 2016 1110:1013:4927 $ hdf5 library version : 1.........10.4 threadsafe exse library version : 1.4.1 FILE library version : 1.8.3 ......... |
If you find the CDO commands below do not work, you can either build CDO yourself, making sure a recent version of grib_api or ecCodes is used, or you can use the workaround below.
...
Code Block | ||
---|---|---|
| ||
cdo -R copy <input grib> <output grib> | ||
Info |
The -R
option
...
only works for GRIB edition 1 (GRIB 1) data as it uses the CGRIBEX decoder.
For GRIB-2 messages it's better to use the setgridtype
operator:
Code Block | ||
---|---|---|
| ||
cdo setgridtype,regular <input grib> <output grib>
cdo -f nc setgridtype,regular <input grib> <output netcdf> |
Info |
---|
If you have files with a mix of GRIB-1 and GRIB-2, then either split the file first or compile cdo with "--disable-cgribex --with-eccodes=yes" If variable names are lost, add the "-t ecmwf" option. |
If these steps do not work, see workarounds below.
Steps to convert GRIB to netCDF
Split z axis
Before converting to netCDF, separate the different vertical axes using the generic command: cdo splitzaxis <input file> <output file pattern>.
This will ensure the following steps work correctly:
Code Block | ||
---|---|---|
| ||
cdo splitzaxis ICMSHg4a4+000000.grb ICMSHg4a4+000000_split
ls ICMSH*split*
ICMSHg4a4+000000_split01.grb
ICMSHg4a4+000000_split02.grb
ICMSHg4a4+000000_split03.grb
ICMSHg4a4+000000_split04.grb |
In this example, the different files might contain: '01' - fields on pressure levels, '02' - fields on model levels, '03' - hybrid levels, '04' - surface and so on. The number of files created depends on the number of different types of levels. Use the command grib_ls
to inspect the contents of each split file.
Note that the filename ICMSH
in this example, indicates it contains spectral and not gridpoint fields.
Another way to split the file would be using the grib_copy command from the ecCodes software as shown above:
Code Block |
---|
grib_copy ICMSHg4a4+000000.grb ICMSHg4a4+000000_[typeOfLevel].grb |
In this case 'typeOfLevel' is a GRIB key. The square brackets is a special syntax to the grib_copy
command. This approach works with any GRIB key.
Convert spectral to gridpoint
CDO supports two options for converting spectral to gridpoint data:
- sp2gpl - converts spectral to a linear Gaussian grid, appropriate for IFS data.
- sp2gp - converts spectral to a quadratic Gaussian grid (only use this for T21/42 resolutions)
Both produce a regular Gaussian grid with equal number of longitudes on each latitude row. Remember that a Gaussian grid has irregularly spaced latitudes (but nearly regular). For more details please see the description of these operators on the CDO homepage.
Code Block |
---|
cdo -f nc -sp2gpl <input spectral coefficent grib> <output gridded netcdf> |
Convert the resulting output files to netcdf
After any operation converting a GRIB file to any other GRIB file, the conversion to netCDF is just:
Code Block | ||
---|---|---|
| ||
cdo -f nc copy <input grib> <output netcdf> |
Convert vorticity and divergence to wind
To convert wind components u & v instead of vorticity and divergence, use:
Code Block | ||
---|---|---|
| ||
cdo dv2uvl <input file> <output file> |
Interpolate to pressure from model levels
To interpolate to pressure levels from model levels, use:
Code Block | ||
---|---|---|
| ||
cdo ml2pl,92500,85000,50000,20000 <input file> <output file> |
Viewing contents of GRIB files
There are various commands for inspecting the contents of a GRIB file. The ecCodes software installation required for OpenIFS has useful commands like grib_ls
and grib_dump
.
Code Block | ||
---|---|---|
| ||
grib_ls ICMSHfrq2+000000.grb
grib_dump ICMSHfrq2+000000.grb |
Here is another example using the cdo command:
Code Block | ||
---|---|---|
| ||
$ cdo sinfo ICMSHfrq2+000000.grb
File format: GRIB
-1 : Institut Source Param |
If this step does not work, see workarounds below.
Steps to convert GRIB to netCDF
Split z axis
Before converting to netCDF, separate the different vertical axes using the generic command: cdo splitzaxis <input file> <output file pattern>.
This will ensure the following steps work correctly:
Code Block | ||
---|---|---|
| ||
cdo splitzaxis ICMSHg4a4+000000.grb ICMSHg4a4+000000_split
ls ICMSH*split*
ICMSHg4a4+000000_split01.grb
ICMSHg4a4+000000_split02.grb
ICMSHg4a4+000000_split03.grb
ICMSHg4a4+000000_split04.grb |
In this example, the different files might contain: '01' - fields on pressure levels, '02' - fields on model levels, '03' - hybrid levels, '04' - surface and so on. The number of files created depends on the number of different types of levels. Use the command grib_ls
to inspect the contents of each split file.
Note that the filename ICMSH
in this example, indicates it contains spectral and not gridpoint fields.
Another way to split the file would be using the grib_copy command from the ecCodes (or grib_api) software as shown above:
Code Block |
---|
grib_copy ICMSHg4a4+000000.grb ICMSHg4a4+000000_[typeOfLevel].grb |
In this case 'typeOfLevel' is a GRIB key. The square brackets is a special syntax to the grib_copy
command. This approach works with any GRIB key.
Convert spectral to gridpoint
CDO supports two options for converting spectral to gridpoint data:
- sp2gpl - converts spectral to a linear Gaussian grid, appropriate for IFS data.
- sp2gp - converts spectral to a quadratic Gaussian grid (only use this for T21/42 resolutions)
Both produce a regular Gaussian grid with equal number of longitudes on each latitude row. Remember that a Gaussian grid has irregularly spaced latitudes (but nearly regular). For more details please see the description of these operators on the CDO homepage.
Convert the resulting output files to netcdf
Code Block | ||
---|---|---|
| ||
cdo -f nc copy <input grib> <output netcdf> |
If you prefer wind components u & v instead of vorticity and divergence, use the cdo command:
Code Block | ||
---|---|---|
| ||
cdo dv2uvl <input file> <output file> |
To interpolate to pressure levels from model levels, use this cdo command, for example:
Code Block | ||
---|---|---|
| ||
cdo ml2pl,92500,85000,50000,20000 <input file> <output file> |
Viewing contents of GRIB files
There are various commands for inspecting the contents of a GRIB file. The GRIB_API installation as part of OpenIFS has useful commands like grib_ls
and grib_dump
.
Code Block | ||
---|---|---|
| ||
grib_ls ICMSHfrq2+000000.grb
grib_dump ICMSHfrq2+000000.grb |
Here is another example using the cdo command:
Code Block | ||
---|---|---|
| ||
$ cdo sinfo ICMSHfrq2+000000.grb
File format: GRIB
-1 : Institut Source Param Time Typ Grid Size Num Levels Num
1 : ECMWF unknown 11.3 var P16 65792 1 60 1
2 : ECMWF unknown 39.3 var P16 65792 1 60 1
3 : ECMWF unknown 43.3 var P16 65792 1 60 1
4 : ECMWF unknown 44.3 var P16 65792 1 60 1
5 : ECMWF unknown 152.128 var P16 65792 1 1 2
6 : ECMWF unknown 6.3 var P16 65792 1 1 2
Horizontal grids :
1 : spectral > size : dim = 65792 truncation = 255 spc = 32896
complexPacking = 1
Vertical grids :
1 : hybrid level : 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51
52 53 54 55 56 57 58 59 60
2 : hybrid level : 1
Time axis : 1 step
RefTime = 1999-12-24 12:00:00 Units = hours Calendar = PROLEPTIC
YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss
1999-12-24 12:00:00
cdo sinfo: Processed 6 variables over 1 timestep. ( 0.04s ) |
...
Code Block | ||
---|---|---|
| ||
for i in 000 024 048 072 do cdo -R copy ICMGGfpp4+000${i}.grb ICMGGfpp4+000${i}_R.grb cdo splitzaxis ICMGGfpp4+000${i}_R.grb ICMGGfpp4+000${i} cdo -f nc copy ICMGGfpp4+000${i}01.grb ICMGGfpp4+000${i}_PL_GP.nc # pressure level gridpoint fields cdo -f nc copy ICMGGfpp4+000${i}02.grb ICMGGfpp4+000${i}_ML_GP.nc # model level gridpoint fields cdo splitzaxis ICMSHfpp4+000${i}.grb ICMSHfpp4+000${i} cdo -f nc copy -sp2gpl ICMSHfpp4+000${i}01.grb ICMSHfpp4+000${i}01_PL_GP.grb # spectral fields on press levels transformed to gridpoint cdo -f nc copy -sp2gpl ICMSHfpp4+000${i}03.grb ICMSHfpp4+000${i}03_ML_GP.grb # spectral fields on model levels transformed to gridpoint cdo -f nc copy -sp2gpl ICMSHfpp4+000${i}04.grb ICMSHfpp4+000${i}04_surf_GP.grb # spectral fields on surface levels transformed to gridpoint done |
Possible problems and solutions
cdo -R option does not work with GRIB-2 fields to convert to regular grid
CDO's -R option, to convert from reduced Gaussian grid to regular Gaussian grid, only works with GRIB 1 format (cgribex only) (see cdo man page). This is a problem as multi-level output from OpenIFS is encoded as GRIB-2 data.
A workaround is to temporarily change the edition number to 1. Use the grib_set
command (from ecCodes or grib_api) to change the GRIB edition number for all messages:
Code Block | ||
---|---|---|
| ||
grib_set -s editionNumber=1 in.grb in_1.grb
cdo -R copy in_1.grb in_1_R.grb
grib_set -s editionNumber=2 in_1_R.grb in_2_R.grb |
This does not actually convert the message to GRIB-1, it only changes the editionNumber. Any GRIB-2 specific elements are unaltered.
It's recommended that the editionNumber
is changed back to 2 to ensure that the parameter names remain correct in any subsequent cdo / grib commands.
Workaround if using more than 128 levels
The workaround above will fail if the grib file contains field with more than 128 levels. This is because there are not enough bits available in the GRIB-1 parameter to encode the total number of model half levels (which would exceed 255).
In this case, the 'pv' GRIB parameter causes the problem (nothing to do with potential vorticity!). This is an array holding the half level values of the model's A & B coefficients that define the location of the levels.
The workaround above can be extended to delete this array while setting the edition number to 1, like this:
Note |
---|
cdo commands can be combined into a single command for greater efficiency. |
Using EMOSLIB to interpolate to regular grid
The ECMWF interpolation software library EMOSLIB also provides the capability to interpolate spectral data to regular gaussian grids or regular lat-lon grids, and interpolate regular gaussian grids to regular lat-lon grids.
The EMOSLIB library provides a Fortran library to enable users to write their own interpolation software. Please see the EMOSLIB website for more details and examples.
EMOSLIB also provides two command-line tools for using interpolation. Assuming a recent version of the EMOS library, there are two tools that can be used.
In the 'bin'
directory there is a command 'emos_tool'
which can be used as:
Code Block | ||
---|---|---|
| ||
emos_tool --regular=256 [--area="40/-10/31/20"] in.grib out.grib |
The grid specified for the --regular
option follows the grid naming convention for EMOSLIB here: Reduced Gaussian Grids.
Another command can be found in the 'tools'
directory. This allows the interpolation function to be specified and is more flexible:
Code Block |
---|
tools/int --INTOUT:gridname=F256 --INTOUT:area="40/-10/31/20" --input=1.grib --output=2.grib --intf2 |
Here 'F256' means 'full-grid' not reduced, and --intf2
is the EMOSLIB interpolation function to be used.
Possible problems and solutions
cdo -R option does not work with GRIB-2 fields to convert to regular grid
CDO's -R option, to convert from reduced Gaussian grid to regular Gaussian grid, only works with GRIB 1 format as the CGRIBEX decoder this uses does not work with GRIB-2 (see cdo man page). This is a problem as multi-level output from OpenIFS is encoded as GRIB-2 data.
The preferred way to deal with this is to use the setgridtype operator as described above:
Code Block |
---|
cdo -f nc setgridtype,regular <input grib> <output netcdf> |
However, if this does not work for any reason, a workaround is to temporarily change the edition number to 1 of the GRIB file. Use the grib_set
command (from ecCodes) to change the GRIB edition number for all messages:
Code Block | ||
---|---|---|
| ||
grib_set -s editionNumber=1 in.grb in_1.grb
cdo -R | ||
Code Block | ||
grib_set -s deletePV=1,editionNumber=1 in.grb in_1.grb
cdo -R copy in_1.grb in_1_R.grb
grib_set -s editionNumber=2 in_1_R.grb in_2_R.grb |
The deletePV option is known as a 'concept' rather than a parameter contained in the GRIB file itself. It ensures the PV array is deleted correctly by grib_set. Now the cdo regridding will work correctly, but remember to reset the editionNumber back to 2.
The A & B half level coefficients can be recovered from the original file by:
Code Block |
---|
cdo vct in.grb > pv.txt |
Parameter names are lost for GRIB-1 fields
ECMWF GRIB-1 use keys which may not be recognised by CDO because they are locally defined (e.g. the shortName
key) and not defined in the WMO GRIB tables that CDO uses. This can cause parameter names to be lost or not recognised when using the CDO commands.
A workaround is to use the -t ecmwf
option. The -t option
tells CDO to use the predefined ECMWF parameter tables (see CDO documentation for more details).
Code Block |
---|
cdo -t ecmwf -f nc copy mygrib1.grb mygrib1.nc |
Warning |
---|
Do not use this option for GRIB-2 fields. It sets the GRIB table default to be specific to the GRIB-1 ECMWF tables. If problems persist, we recommend using grib_to_netcdf to convert to netCDF. |
Acknowledgements
Thanks to Paul Dando of User Support for help with the contents of this page.
...
This does not actually convert the message to GRIB-1, it only changes the editionNumber. Any GRIB-2 specific elements are unaltered.
It's recommended that the editionNumber
is changed back to 2 to ensure that the parameter names remain correct in any subsequent cdo / grib commands.
Workaround if using more than 128 levels
The workaround above will fail if the grib file contains field with more than 128 levels. This is because there are not enough bits available in the GRIB-1 parameter to encode the total number of model half levels (which would exceed 255).
In this case, the 'pv' GRIB parameter causes the problem (nothing to do with potential vorticity!). This is an array holding the half level values of the model's A & B coefficients that define the location of the levels.
The workaround above can be extended to delete this array while setting the edition number to 1, like this:
Code Block |
---|
grib_set -s deletePV=1,editionNumber=1 in.grb in_1.grb
cdo -R copy in_1.grb in_1_R.grb
grib_set -s editionNumber=2 in_1_R.grb in_2_R.grb |
The deletePV option is known as a 'concept' rather than a parameter contained in the GRIB file itself. It ensures the PV array is deleted correctly by grib_set. Now the cdo regridding will work correctly, but remember to reset the editionNumber back to 2.
The A & B half level coefficients can be recovered from the original file by:
Code Block |
---|
cdo vct in.grb > pv.txt |
Parameter names are lost for GRIB-1 fields
ECMWF GRIB-1 use keys which may not be recognised by CDO because they are locally defined (e.g. the shortName
key) and not defined in the WMO GRIB tables that CDO uses. This can cause parameter names to be lost or not recognised when using the CDO commands.
A workaround is to use the -t ecmwf
option. This will make CDO use ECMWF parameter table 128 for the definitions of variables but 128 is only for GRIB 1. This will work for the surface fields which are encoded by IFS as GRIB 1, but not for the multi-level fields which are encoded in GRIB 2. Variable names become 'unknown' if -t ecmwf is used with the multi-level fields. It is best to omit it completely, even for surface fields, and only use it for GRIB 1 data if you get missing variable names after CDO has converted the data (see CDO documentation for more details).
Code Block |
---|
cdo -t ecmwf -f nc copy mygrib1.grb mygrib1.nc |
Warning |
---|
Do not use this option for GRIB-2 fields. It sets the GRIB table default to be specific to the GRIB-1 ECMWF tables. If problems persist, we recommend using grib_to_netcdf to convert to netCDF. |
Acknowledgements
Thanks to Paul Dando of User Support for help with the contents of this page.
HTML Comment | ||
---|---|---|
| ||
The excerpt below includes a Google analytics tracker, do not remove! |
Excerpt Include | ||||||
---|---|---|---|---|---|---|
|
...