You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 11 Current »


Documentation

The ecCodes GRIB tools are documented at:

There are examples of how to use them at:

Other useful resources:

Preliminaries

If you have already followed the Introduction Tutorial on GRIB decoding with ecCodes GRIB tools: inspecting GRIB files there is no need to redo these preliminary steps.

Login to ecs-login and work in your $SCRATCH:

$> cd $SCRATCH

Make a copy of the practicals directory in your $SCRATCH to Atos:

$> tar –xvf /home/trx/grib_decoding.tar

This will create a directory in your $SCRATCH called grib_decoding containing the GRIB data files for this tutorial.

There are sub-directories for each practical:

$> ls $SCRATCH/grib_decoding

inspect  modify  python

For this tutorial, cd to the modify directory:

$> cd $SCRATCH/grib_decoding/modify


Remember to load the ecmwf-toolbox !

$> module load ecmwf-toolbox

This tutorial covers the following topics:

Using grib_copy to copy selected messages from a GRIB file

The file tq_an_pl.grib1 contains parameters T and Q on six pressure levels.

How could you use grib_copy to create two files, one containing all of the pressure levels for parameter T, the other for parameter Q ?  Check the content of the new files with grib_ls.

To copy only those messages for parameter T from the GRIB file tq_an_pl.grib2 use grib_copy with the -w option to specify a shortName=t:

$>  grib_copy -w shortName=t tq_an_pl.grib2 t_an_pl.grib2

Similarly to copy only those messages for parameter Q, use:

$> grib_copy -w shortName=q tq_an_pl.grib2 q_an_pl.grib2

Or, more simply, and because the file contains only parameters T and Q, one can use:

$> grib_copy tq_an_pl.grib2 "[shortName]_an_pl.grib2"

Using grib_ls of the two files confirms that the contents are correct. For example:

$> grib_ls -p centre,paramId,typeOfLevel,level,date,dataType,time,shortName t_an_pl.grib2
t_an_pl.grib2
centre       paramId      typeOfLevel  level        date         dataType     time         shortName    
ecmf         130          isobaricInhPa  850          20251013     fc           0            t           
ecmf         130          isobaricInhPa  300          20251013     fc           0            t           
ecmf         130          isobaricInhPa  1000         20251013     fc           0            t           
ecmf         130          isobaricInhPa  700          20251013     fc           0            t           
ecmf         130          isobaricInhPa  500          20251013     fc           0            t           
ecmf         130          isobaricInhPa  400          20251013     fc           0            t           
6 of 6 messages in t_an_pl.grib2

6 of 6 total messages in 1 files


You can take this further and split the file tq_an_pl.grib1 into separate files for each parameter/pressure level combination with:

$> grib_copy tq_an_pl.grib2 "[shortName]_[level].grib[edition]"

$> ls ?_*0.grib?
q_1000.grib2  q_400.grib2  q_700.grib2	q_an_pl.grib2  t_300.grib2  t_500.grib2  t_850.grib2
q_300.grib2   q_500.grib2  q_850.grib2	t_1000.grib2   t_400.grib2  t_700.grib2

Each file contains one message only, e.g.:

$> grib_ls -p centre,paramId,typeOfLevel,level,date,dataType,time,shortName t_1000.grib2
t_1000.grib2
centre       paramId      typeOfLevel  level        date         dataType     time         shortName    
ecmf         130          isobaricInhPa  1000         20251013     fc           0            t           
1 of 1 messages in t_1000.grib2

1 of 1 total messages in 1 files

Using grib_set to change key values

The field in the file tp.grib is supposed to contain total precipitation, created using MARS compute to sum the convective and large-scale (stratiform) precipitation.

What is the parameter set to in the file tp.grib2 ?   How could you use grib_set to change it is set correctly to total precipitation (shortName=tp, paramId=228) ?

Use grib_ls to inspect the contents of the file tp.grib2, printing also the paramId with the –P option:

$> grib_ls -P paramId tp.grib2
tp.grib2
paramId      edition      centre       date         dataType     gridType     typeOfLevel  level        stepRange    shortName    packingType  
142          2            ecmf         20241216     fc           regular_ll   surface      0            0-24         lsp          grid_simple 
1 of 1 messages in tp.grib2

1 of 1 total messages in 1 files

Alternatively, you could inspect the parameter namespace with grib_ls:

$> grib_ls -n parameter tp.grib2
tp.grib2
centre      paramId     shortName   units       name        
ecmf        142         lsp         m           Large-scale precipitation 
1 of 1 messages in tp.grib2

1 of 1 total messages in 1 files

The shortName is set to lsp and paramId to 142, i.e., large-scale precipitation.

The field was created using MARS compute to sum the convective and large-scale (stratiform) precipitation. When using MARS compute, the output field maintains the information from the GRIB header of the first input field so, in this case, we see that the field has paramId 142 even though the message is now supposed to contain the total precipitation.

To change the parameter to total precipitation, use grib_set as follows:

$> grib_set -s shortName=tp tp.grib2 tp_new.grib2

Check with grib_ls:

$> grib_ls -n parameter tp_new.grib2 
tp_new.grib2
centre      paramId     shortName   units       name        
ecmf        228         tp          m           Total precipitation 
1 of 1 messages in tp_new.grib2

1 of 1 total messages in 1 files

Note that changing the shortName has also changed the value of paramId. The same could be achieved by setting paramId=228:

$> grib_set -s paramId=228 tp.grib2 tp_new.grib2

Converting GRIB data to netCDF with grib_to_netcdf

How could you use the grib_to_netcdf tool to convert the GRIB messages in file1.grib to netCDF and store the result in a file named out1.nc using the default NC_SHORT netCDF data type ?


To convert the GRIB messages in file.grib2 to netCDF with data type NC_SHORT and store in a file named out1.nc use:

$> grib_to_netcdf -o out1.nc file1.grib2
grib_to_netcdf: Version 2.44.0
grib_to_netcdf: Processing input file 'file1.grib2'.
grib_to_netcdf: Found 5 GRIB fields in 1 file.
grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
grib_to_netcdf: Creating netCDF file 'out1.nc'
grib_to_netcdf: NetCDF library version: 4.9.3 of Mar 14 2025 07:27:22 $
grib_to_netcdf: Creating large (64 bit) file format.
grib_to_netcdf: Defining variable 't2m'.
grib_to_netcdf: Done.

Load the netcdf4 module and use ncdump to check the values in the netCDF file created in the previous step as follows:

$> module load netcdf4

$> ncdump -v t2m out1.nc

What are the data values set to in the out1.nc netCDF file ?

Using ncdump to print the values for the variable t2m shows:

$> module load netcdf4

$> ncdump -v t2m out1.nc 
netcdf out1 {
...
data:

 t2m =
  -29628, -28125,
  21957, 21552,
  -15905, -32766,
  23701, 21176,
  28426, 32767,
  25246, 23499,
  10705, 4529,
  27955, 29374,
  -32625, -32683,
  30518, 29616 ;
}

The data values appear as integers because of the NC_SHORT data format. These values need to be unpacked using the scale_factor and add_offset netCDF attributes.

How could you change the the grib_to_netcdf command to create a netCDF output with the values stored as floating point numbers ?

To store the values in the netCDF file as floating point numbers, use the grib_to_netcdf tool with the data format to NC_FLOAT or NC_DOUBLE with the -D option.  For example:

$> grib_to_netcdf -D NC_FLOAT -o out2.nc file1.grib2
grib_to_netcdf: Version 2.44.0
grib_to_netcdf: Processing input file 'file1.grib2'.
grib_to_netcdf: Found 5 GRIB fields in 1 file.
grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
grib_to_netcdf: Creating netCDF file 'out2.nc'
grib_to_netcdf: NetCDF library version: 4.9.3 of Mar 14 2025 07:27:22 $
grib_to_netcdf: Creating large (64 bit) file format.
grib_to_netcdf: Defining variable 't2m'.
grib_to_netcdf: Done.

Checking with ncdump, we see the values are now stored as floating point values:

ncdump -v t2m out2.nc                            
netcdf out2 {
...
data:

 t2m =
  282.9801, 283.1394,
  288.4472, 288.4043,
  284.4345, 282.6475,
  288.632, 288.3644,
  289.1328, 289.5929,
  288.7958, 288.6107,
  287.2547, 286.6001,
  289.0829, 289.2333,
  282.6624, 282.6563,
  289.3545, 289.2589 ;
}

In this case, the data values do not need further processing.

Try using grib_to_netcdf to convert the GRIB messages in file2.grib to netCDF.   What happens....and why ?   How would you fix it ?

Converting file2.grib2 to netCDF gives the following error:

$> grib_to_netcdf -o out2.nc file2.grib2            
grib_to_netcdf: Version 2.44.0
grib_to_netcdf: Processing input file 'file2.grib2'.
grib_to_netcdf: Found 5 GRIB fields in 1 file.
grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
grib_to_netcdf: Creating netCDF file 'out3.nc'
grib_to_netcdf: NetCDF library version: 4.9.3 of Mar 14 2025 07:27:22 $
grib_to_netcdf: Creating large (64 bit) file format.
ECCODES ERROR   :  Grid type = reduced_gg
ECCODES ERROR   :  First GRIB is not on a regular lat/lon grid or on a regular Gaussian grid. Exiting.

The conversion fails because the GRIB data is represented on a reduced Gaussian grid (gridType=reduced_gg). Conversion to netCDF is possible only for GRIB data with gridType=regular_ll or gridType=regular_gg. 

To fix this it would be necessary to re-retrieve the data on a regular grid or interpolate to a regular grid.



  • No labels