You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Current »


Documentation

The ecCodes GRIB tools are documented at:

There are examples of how to use them at:

Other useful resources:

Preliminaries

If you have already followed the Introduction Tutorial on GRIB decoding with ecCodes GRIB tools: inspecting GRIB files there is no need to redo these preliminary steps.

Login to ecs-login and work in your $SCRATCH:

$> cd $SCRATCH

Make a copy of the practicals directory in your $SCRATCH to Atos:

$> tar –xvf /home/trx/grib_practicals.tar

This will create a directory in your $SCRATCH called grib_practicals containing the GRIB data files for this tutorial.

There are sub-directories for each practical:

$> ls $SCRATCH/grib_practicals

inspect  modify  python

For this tutorial, cd to the modify directory:

$> cd $SCRATCH/grib_practicals/modify


Remember to load the ecmwf-toolbox !

$> module load ecmwf-toolbox

This tutorial covers the following topics:

Using grib_copy to copy selected messages from a GRIB file

The file tz_an_pl.grib1 contains parameters T and Z on six pressure levels.

How could you use grib_copy to create two files, one containing all of the pressure levels for parameter T, the other for parameter Z ?  Check the content of the new files with grib_ls.

To copy only those messages for parameter T from the GRIB file tz_an_pl.grib1 use grib_copy with the -w option to specify a shortName=t:

$>  grib_copy -w shortName=t tz_an_pl.grib1 t_an_pl.grib1

Similarly to copy only those messages for parameter Z, use:

$> grib_copy -w shortName=z tz_an_pl.grib1 z_an_pl.grib1

Or, more simply, and because the file contains only parameters T and Z, one can use:

$> grib_copy tz_an_pl.grib1 "[shortName]_an_pl.grib1"

Using grib_ls of the two files confirms that the contents are correct. For example:

$> grib_ls -p centre,paramId,typeOfLevel,level,date,dataType,time,shortName t_an_pl.grib1
t_an_pl.grib1
centre       paramId      typeOfLevel  level        date         dataType     time         shortName    
ecmf         130          isobaricInhPa  1000         20170219     an           1200         t           
ecmf         130          isobaricInhPa  850          20170219     an           1200         t           
ecmf         130          isobaricInhPa  700          20170219     an           1200         t           
ecmf         130          isobaricInhPa  500          20170219     an           1200         t           
ecmf         130          isobaricInhPa  400          20170219     an           1200         t           
ecmf         130          isobaricInhPa  300          20170219     an           1200         t           
6 of 6 messages in t_an_pl.grib1
  
6 of 6 total messages in 1 files


You can take this further and split the file tz_an_pl.grib1 into separate files for each parameter/pressure level combination with:

$> grib_copy tz_an_pl.grib1 "[shortName]_[level].grib[edition]"

$> ls ?_*.grib1
t_1000.grib1  t_400.grib1  t_700.grib1  z_1000.grib1  z_400.grib1  z_700.grib1
t_300.grib1   t_500.grib1  t_850.grib1  z_300.grib1   z_500.grib1  z_850.grib1

Each file contains one message only, e.g.:

$> grib_ls -p centre,paramId,typeOfLevel,level,date,dataType,time,shortName t_1000.grib1
t_1000.grib1
centre       paramId      typeOfLevel  level        date         dataType     time         shortName    
ecmf         130          isobaricInhPa  1000         20170219     an           1200         t           
1 of 1 messages in t_1000.grib1

1 of 1 total messages in 1 files

Using grib_set to change key values

The field in the file tp.grib is supposed to contain total precipitation, created using MARS compute to sum the convective and large-scale (stratiform) precipitation.

What is the parameter set to in the file tp.grib ?   How could you use grib_set to change it is set correctly to total precipitation (shortName=tp, paramId=228) ?

Use grib_ls to inspect the contents of the file tp.grib1, printing also the paramId with the –P option:

$> grib_ls -P paramId tp.grib1
tp.grib1
paramId      edition      centre       typeOfLevel  level        dataDate     stepRange    dataType     shortName    packingType  gridType     
142          1            ecmf         surface      0            20170221     24           fc           lsp          grid_simple  regular_ll  
1 of 1 messages in tp.grib1

1 of 1 total messages in 1 files

Alternatively, you could inspect the parameter namesapce with grib_ls:

$> grib_ls -n parameter tp.grib1
tp.grib1
centre      paramId     units       name        shortName   
ecmf        142         m           Large-scale precipitation  lsp        
1 of 1 messages in tp.grib1

1 of 1 total messages in 1 files

The shortName is set to lsp and paramId to 142, i.e., large-scale precipitation.

The field was created using MARS compute to sum the convective and large-scale (stratiform) precipitation. When using MARS compute, the output field maintains the information from the GRIB header of the first input field so, in this case, we see that the field has paramId 142 even though the message is now supposed to contain the total precipitation.

To change the parameter to total precipitation, use grib_set as follows:

$> grib_set -s shortName=tp tp.grib1 tp_new.grib1

Check with grib_ls:

$> grib_ls -P paramId tp_new.grib1               
tp_new.grib1
paramId      edition      centre       typeOfLevel  level        dataDate     stepRange    dataType     shortName    packingType  gridType     
228          1            ecmf         surface      0            20170221     24           fc           tp           grid_simple  regular_ll  
1 of 1 messages in tp_new.grib1
   
1 of 1 total messages in 1 files

Note that changing the shortName has also changed the value of paramId. The same could be achieved by setting paramId=228:

$> grib_set -s paramId=228 tp.grib tp_new.grib

Converting GRIB data to netCDF with grib_to_netcdf

How could you use the grib_to_netcdf tool to convert the GRIB messages in file1.grib to netCDF and store the result in a file named out1.nc using the default NC_SHORT netCDF data type ?


To convert the GRIB messages in file.grib1 to netCDF with data type NC_SHORT and store in a file named out1.nc use:

$> grib_to_netcdf -o out1.nc file1.grib1
grib_to_netcdf: Version 2.30.2
grib_to_netcdf: Processing input file 'file1.grib1'.
grib_to_netcdf: Found 4 GRIB fields in 1 file.
grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
grib_to_netcdf: Creating netCDF file 'out1.nc'
grib_to_netcdf: NetCDF library version: 4.9.1 of Feb  9 2023 13:54:09 $
grib_to_netcdf: Creating large (64 bit) file format.
grib_to_netcdf: Defining variable 't2m'.
grib_to_netcdf: Done.

Load the netcdf4 module and use ncdump to check the values in the netCDF file created in the previous step as follows:

$> module load netcdf4

$> ncdump -v t2m out1.nc

What are the data values set to in the out1.nc netCDF file ?

Using ncdump to print the values for the variable t2m shows:

$> module load netcdf4

$> ncdump -v t2m out1.nc 
netcdf out1 {
...
data:

 t2m =
  20497, -809,
  -16312, -32766,
  32767, 7042,
  -1312, -18041,
  17168, -12207,
  -4148, -28392,
  19382, -10827,
  -2506, -25406 ;
}

The data values appear as integers because of the NC_SHORT data format. These values need to be unpacked using the scale_factor and add_offset NetCDF attributes.

How could you change the the grib_to_netcdf command to create a netCDF output with the values stored as floating point numbers ?

To store the values in the netCDF file as floating point numbers, use the grib_to_netcdf tool with the data format to NC_FLOAT or NC_DOUBLE with the -D option.  For example:

$> grib_to_netcdf -D NC_FLOAT -o out2.nc file1.grib1
grib_to_netcdf: Version 2.30.2
grib_to_netcdf: Processing input file 'file1.grib1'.
grib_to_netcdf: Found 4 GRIB fields in 1 file.
grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
grib_to_netcdf: Creating netCDF file 'out2.nc'
grib_to_netcdf: NetCDF library version: 4.9.1 of Feb  9 2023 13:54:09 $
grib_to_netcdf: Creating large (64 bit) file format.
grib_to_netcdf: Defining variable 't2m'.
grib_to_netcdf: Done.

Checking with ncdump, we see the values are now stored as floating point values:

ncdump -v t2m out2.nc                            
netcdf out2 {
...
data:

 t2m =
  281.1666, 277.9943,
  275.6859, 273.236,
  282.9937, 279.1632,
  277.9194, 275.4285,
  280.671, 276.2971,
  277.4971, 273.8872,
  281.0007, 276.5026,
  277.7416, 274.3318 ;
}

In this case, the data values do not need further processing.

Try using grib_to_netcdf to convert the GRIB messages in file2.grib to netCDF.   What happens....and why ?   Can you fix it ?

Converting file2.grib to netCDF gives the following error:

grib_to_netcdf -o out2.nc file2.grib1            
grib_to_netcdf: Version 2.30.2
grib_to_netcdf: Processing input file 'file2.grib1'.
grib_to_netcdf: Found 1 GRIB field in 1 file.
grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
grib_to_netcdf: Creating netCDF file 'out2.nc'
grib_to_netcdf: NetCDF library version: 4.9.1 of Feb  9 2023 13:54:09 $
grib_to_netcdf: Creating large (64 bit) file format.
ECCODES ERROR   :  Grid type = reduced_gg
ECCODES ERROR   :  First GRIB is not on a regular lat/lon grid or on a regular Gaussian grid. Exiting.

The conversion fails because the GRIB data is represented on a reduced Gaussian grid (gridType=reduced_gg). Conversion to NetCDF is possible only for GRIB data with gridType=regular_ll or gridType=regular_gg. 

To fix this it would be necessary to re-retrieve the data on a regular grid or interpolate to a regular grid.



  • No labels