Documentation

The ecCodes GRIB tools are documented at:

ecCodes GRIB tools

There are examples of how to use them at:

ecCodes GRIB tools examples

Other useful resources:

Preliminaries

Work in your $SCRATCH

$> cd $SCRATCH

Make a copy of the practicals directory in your $SCRATCH to Atos:

$> tar –xvf /home/trx/grib_practicals.tar

This will create a directory in your $SCRATCH called grib_practicals containing the GRIB data files for this tutorial.

There are sub-directories for each practical:

$> ls $SCRATCH/grib_practicals

inspect  modify  python

Remember to load the ecmwf-toolbox !

$> module load ecmwf-toolbox

This tutorial covers the following topics:

Using grib_dump and grib_ls to inspect GRIB files

Use grib_ls to inspect the content of the files t2m.grib1 and t2m.grib2

What fields do the GRIB messages contain ?

To list the GRIB messages in t2m.grib1

   $>  grib_ls t2m.grib1
   t2m.grib1
   edition      centre       typeOfLevel  level        dataDate     stepRange    dataType     shortName    packingType  gridType     
   1            ecmf         surface      0            20170217     0            cf           2t           grid_simple  reduced_gg  
   1            ecmf         surface      0            20170217     6            cf           2t           grid_simple  reduced_gg  
   1            ecmf         surface      0            20170217     12           cf           2t           grid_simple  reduced_gg  
   1            ecmf         surface      0            20170217     18           cf           2t           grid_simple  reduced_gg  
   1            ecmf         surface      0            20170217     24           cf           2t           grid_simple  reduced_gg  
   5 of 5 grib messages in t2m.grib1
   
   5 of 5 total grib messages in 1 file

The file contains 2 metre temperature (shortName=2tl) from the ECMWF Ensemble control forecast (dataType=cf) for 6-hourly steps (stepRange) from 0 to 24 hours for dataDate=20170217.

Similarly, for t2m.grib2:

$>  grib_ls t2m.grib2
t2m.grib2
edition      centre       date         dataType     gridType     stepRange    typeOfLevel  level        shortName    packingType  
2            ecmf         20170217     cf           reduced_gg   0            heightAboveGround  2            2t           grid_simple 
2            ecmf         20170217     cf           reduced_gg   6            heightAboveGround  2            2t           grid_simple 
2            ecmf         20170217     cf           reduced_gg   12           heightAboveGround  2            2t           grid_simple 
2            ecmf         20170217     cf           reduced_gg   18           heightAboveGround  2            2t           grid_simple 
2            ecmf         20170217     cf           reduced_gg   24           heightAboveGround  2            2t           grid_simple 
5 of 5 grib messages in t2m.grib1
   
5 of 5 total grib messages in 1 files

Again, the file contains 2 metre temperature (shortName=2t) from the ECMWF Ensemble control forecast (dataType=cf) for 6-hourly steps (stepRange) from 0 to 24 hours for dataDate=20170217.

In fact, they are the same fields as in t2m.grib1 but encoded in GRIB edition 2 and obtained from the TIGGE archive.

Do the default keys printed by grib_ls depend on the GRIB edition ? Do you see any other differences ?

No, grib_ls prints the same set of default keys for the both GRIB edition 1 and GRIB edition2. However, the order of the columns output is different.

The only differences seen with grib_ls is for the value of the typeOfLevel and level keys which are

typeOfLevel=surface and level=0 inthe GRIB1 encoding; and
typeOfLevel=heightAboveGround and level=2 for the GRIB2 encoding.

How do you print the MARS keys for the two files ?

To print the mars keys, use either grib_ls -m or grib_ls -n mars. This is the output for the file t2m.grib1:

$> grib_ls -m t2m.grib1
t2m.grib1
domain      levtype     date        time        step        param       class       type        stream      expver      
g           sfc         20170217    0000        0           167.128     od          cf          enfo        0001       
g           sfc         20170217    0000        6           167.128     od          cf          enfo        0001       
g           sfc         20170217    0000        12          167.128     od          cf          enfo        0001       
g           sfc         20170217    0000        18          167.128     od          cf          enfo        0001       
g           sfc         20170217    0000        24          167.128     od          cf          enfo        0001       
5 of 5 grib messages in t2m.grib1

5 of 5 total grib messages in 1 files

For t2m.grib2, the same command gives:

$> grib_ls -m t2m.grib2
t2m.grib2
origin      date        time        step        levtype     number      param       expver      class       model       type        stream      
ecmf        20170217    0000        0           sfc         0           167         prod        ti          glob        cf          enfo       
ecmf        20170217    0000        6           sfc         0           167         prod        ti          glob        cf          enfo       
ecmf        20170217    0000        12          sfc         0           167         prod        ti          glob        cf          enfo       
ecmf        20170217    0000        18          sfc         0           167         prod        ti          glob        cf          enfo       
ecmf        20170217    0000        24          sfc         0           167         prod        ti          glob        cf          enfo       
5 of 5 grib messages in t2m.grib2
   
5 of 5 total grib messages in 1 files

How do you add the shortName to the output of the previous command ?

To print the shortName along with the list of mars keys use the –P option e.g.:

$> grib_ls -m -P shortName t2m.grib1
t2m.grib1
shortName   domain      levtype     date        time        step        param       class       type        stream      expver      
2t          g           sfc         20170217    0000        0           167.128     od          cf          enfo        0001       
2t          g           sfc         20170217    0000        6           167.128     od          cf          enfo        0001       
2t          g           sfc         20170217    0000        12          167.128     od          cf          enfo        0001       
2t          g           sfc         20170217    0000        18          167.128     od          cf          enfo        0001       
2t          g           sfc         20170217    0000        24          167.128     od          cf          enfo        0001       
5 of 5 grib messages in t2m.grib1

5 of 5 total grib messages in 1 files

How do you change the previous command so that the output is in descending step order ?

Hint: think about stings and integers

To order the output in descending step order use the –B option:

$> grib_ls -B "step:i desc" t2m.grib1
edition      centre       typeOfLevel  level        dataDate     stepRange    dataType     shortName    packingType  gridType     
1            ecmf         surface      0            20170217     24           cf           2t           grid_simple  reduced_gg  
1            ecmf         surface      0            20170217     18           cf           2t           grid_simple  reduced_gg  
1            ecmf         surface      0            20170217     12           cf           2t           grid_simple  reduced_gg  
1            ecmf         surface      0            20170217     6            cf           2t           grid_simple  reduced_gg  
1            ecmf         surface      0            20170217     0            cf           2t           grid_simple  reduced_gg  
5 of 5 grib messages in t2m.grib1

5 of 5 total grib messages in 1 files

Note that you need to specify “step:i” so that step is treated as an integer rather than a character string.

How do you use grib_ls to print only the centre, dataDate, stepRange, typeOfLevel and shortName ?

To specify a set of keys to print, use the -p option:

$> grib_ls -p centre,dataDate,stepRange,typeOfLevel,shortName 2tm.grib1
t2m.grib1
centre       dataDate     stepRange    typeOfLevel  shortName    
ecmf         20170217     0            surface      t2m         
ecmf         20170217     6            surface      t2m         
ecmf         20170217     12           surface      t2m         
ecmf         20170217     18           surface      t2m         
ecmf         20170217     24           surface      t2m         
5 of 5 grib messages in t2m.grib1

5 of 5 total grib messages in 1 files

How do you change the previous command so that only the field for forecast step=6 is printed ?

To print the keys for forecast step 6 only, use the -p option with the -w option:

$> grib_ls -w stepRange=6 -p centre,dataDate,stepRange,typeOfLevel,shortName t2m.grib1
t2m.grib1
centre      date        stepRange   levelType   shortName
ecmf         20170217     6            surface      t2m
1 of 5 grib messages in t2m.grib1
  
1 of 5 total grib messages in 1 files

Using the same commands with the file t2m.grib2 gives similar output.

How do you change the previous command to print the centre as both a string and an integer ?

Specifying the key as "centre:i" and “centre:s” prints the centre both as an integer and a string:

$> grib_ls -w stepRange=6 -p centre:l,centre:s,dataDate,stepRange,typeOfLevel,shortName t2m.grib1
t2m.grib1
centre       centre       dataDate     stepRange    typeOfLevel  shortName    
98           ecmf         20170217     6            surface      t2m        
1 of 5 grib messages in t2m.grib1

1 of 5 total grib messages in 1 files

As before, using the same commands with the file t2m.grib2 gives similar output.

How do you use grib_dump to inspect the fourth (count=4) GRIB message in both files in WMO Octet mode ?

To use grib_dump to inspect the contents of the fourth message in t2m.grib1 and t2m.grib2 use the -w option with "count=4" (the count key counts the number of messages starting form count=1 for the first message. To print the output in WMO Octet mode use the -O option:

$> grib_dump –w count=4 -O t2m.grib1
...
$> grib_dump –w count=4 -O t2m.grib2
...

The command may produce many lines of output. You may want to redirect output to a file using, e.g.:

$> grib_dump .... t2m.grib1 > t2m.grib1.output

To use grib_dump to all of the keys for the fourth message in JSON mode, use the -j option:

$> grib_dump -j -w count=4 t2m.grib1 > t2m.grib1.output

$> head -20 t2m.grib1.output
{ "messages" : [
  [

    {
      "key" : "editionNumber",
      "value" : 1
    },
    {
      "key" : "table2Version",
      "value" : 128
    },
    {
      "key" : "centre",
      "value" : 98
    },
    {
      "key" : "generatingProcessIdentifier",
      "value" : 147
    },
    {

Similarly, for t2m.grib2:

$> grib_dump -j -w count=4 t2m.grib2 > t2m.grib2.output

$> head -20 t2m.grib1.output
{ "messages" : [
  [

    {
      "key" : "discipline",
      "value" : 0
    },
    {
      "key" : "editionNumber",
      "value" : 2
    },
    {
      "key" : "centre",
      "value" : 98
    },
    {
      "key" : "subCentre",
      "value" : 0
    },
    {

Using grib_ls to find values at the nearest grid point to a specified latitude-longitude point

The file t2m.grib1 contains the 2m temperature from the ENS control forecast at 6-hourly time steps for the first 24 hours on the O640 octahedral reduced Gaussian grid. The file lsm.grib1 contains the ENS land-sea mask on the O640 grid.

How do you use grib_ls to find the value of the 2m temperature at the grid point nearest to ECMWF's headquarters in the UK (Lat 51.42°N, Lon 0.95°W) at each forecast step ?

Be careful to specify the longitude correctly !

To list the nearest points to Lat 51.42° N, Lon 0.95° W use:

$> grib_ls -l 51.42,-0.95 t2m.grib1
t2m.grib1
...
5 of 5 grib messages in t2m.grib1

5 of 5 total grib messages in 1 files
Input Point: latitude=51.42  longitude=-0.95
Grid Point chosen #2 index=156197 latitude=51.38 longitude=359.03 distance=4.79 (Km)
Other grid Points
- 1 - index=156198 latitude=51.38 longitude=359.35 distance=21.64 (Km)   
- 2 - index=156197 latitude=51.38 longitude=359.03 distance=4.79 (Km)
- 3 - index=155082 latitude=51.52 longitude=359.35 distance=23.65 (Km)
- 4 - index=155081 latitude=51.52 longitude=359.03 distance=11.09 (Km)

Note a longitude of 0.95° W must be specified as -0.95 !

Note that ecCodes converts all longitude values so that they lie in the interval 0° to 360°. In GRIB 2 all longitude values must be positive !

From the output of the previous command, what is the latitude and longitude of the grid point nearest to ECMWF's Headquarters in Reading, UK ?

The nearest grid point is at latitude=51.38° longitude=359.03° (0.77° W) and is 4.79 km from ECMWF.

How do you change the previous command to output only the forecast step and the 2m temperature value at the grid point nearest to ECMWF ?

To output only the forecast step and 2 metre temperature value at the nearest grid point set the MODE option to 1 and specify that the stepRange key only should be printed using the -p option:

$> grib_ls -l 51.42,-0.95,1 -p stepRange t2m.grib1
t2m.grib1
stepRange    value 
0           279.889     
6           278.54      
12          282.242     
18          278.133     
24          276.823
5 of 5 messages in t2m.grib1
   
5 of 5 total messages in 1 files
Input Point: latitude=51.42  longitude=-0.95
Grid Point chosen #2 index=156197 latitude=51.38 longitude=359.03 distance=4.79 (Km)
Other grid Points
- 1 - index=156198 latitude=51.38 longitude=359.35 distance=21.64 (Km)
- 2 - index=156197 latitude=51.38 longitude=359.03 distance=4.79 (Km)
- 3 - index=155082 latitude=51.52 longitude=359.35 distance=23.65 (Km)
- 4 - index=155081 latitude=51.52 longitude=359.03 distance=11.09 (Km)

How would you change the previous command to specific that the land-sea mask in the lsm.grib1 file is used to identify the nearest land point and print the 2m temperature values at the nearest grid point in decimal format with 2 decimal places ?

The output format of the values is specified with the -F option. Here, -F"%.2f" is used to specify decimal format with 2 decimal places.

To specify the land-sea mask, pass the name of the file containing the land-sea mask as the FILE option:

% grib_ls -F"%.2f" -l 51.42,-0.95,1,lsm.grib1 -p step  t2m.grib1 
t2m.grib1
step         value 
0           279.89      
6           278.54      
12          282.24      
18          278.13      
24          276.82      
5 of 5 messages in t2m.grib1
   
5 of 5 total messages in 1 files
Input Point: latitude=51.42  longitude=-0.95
Grid Point chosen #2 index=156197 latitude=51.38 longitude=359.03 distance=4.79 (Km)
Mask values:
- 1 - index=156198 latitude=51.38 longitude=359.35 distance=21.64 (Km) value=1.00
- 2 - index=156197 latitude=51.38 longitude=359.03 distance=4.79 (Km) value=0.99
- 3 - index=155082 latitude=51.52 longitude=359.35 distance=23.65 (Km) value=0.99
- 4 - index=155081 latitude=51.52 longitude=359.03 distance=11.09 (Km) value=0.99

Here the value=1.00 (or 0.99) at the end of the line showing the four nearest grid points indicates which are land points (mask=1.0). Land points have mask ≥ 0.5; sea points have mask<0.5.

Once the index of the nearest grid point is known, its value can be obtained directly with the -i option:

$> grib_ls -F"%.2f" -i 156197 -p step  t2m.grib1                 
t2m.grib1
step                value(156197) 
0          279.89      
6          278.54      
12         282.24      
18         278.13      
24         276.82      
5 of 5 messages in t2m.grib1
   
5 of 5 total messages in 1 files

Using grib_get_data to print data values

How do you print the latitude, longitude and values for the field in surface1.grib with the values output in decimal format with 5 decimal places ?

To print the data values for surface.grib1 in decimal format with 6 decimal places use grib_get_data with the option -F"%.6f":

$> grib_get_data -F"%.6f" surface.grib1
Latitude Longitude Value
    2.500  -20.000 301.848755
    2.500  -17.500 302.186279
    2.500  -15.000 301.810547
    2.500  -12.500 301.721924
    ...

How do you change the previous command to also output the latitude and longitude with 6 decimal places ?

To output also the latitude and longitude values with 6 decimal places, we need to use the -L option:

grib_get_data -L"%.6f %.6f" -F"%.6f" surface.grib1
Latitude Longitude Value
2.500000 -20.000000 301.848755
2.500000 -17.500000 302.186279
2.500000 -15.000000 301.810547
2.500000 -12.500000 301.721924
...

Note that we need to specify the format for both latitude and longitude separately: -L"%.6f %.6f".

How can you find out whether there are any missing values in the field in surface1.grib ?

Missing values can be identified by using grib_get_data with the –m option, for example:

$> grib_get_data –m MISSING surface.grib1
Latitude, Longitude, Value
...
  2.500    7.500 3.0176782227e+02
  2.500   10.000 MISSING
  0.000  -20.000 3.0134069824e+02
...

You can also print the key numberOfMissing with the grib_get or grib_ls tools:

$> grib_get -p numberOfMissing surface.grib1
3

So there are 3 missing values in total.

Other keys also provide the numberOfPoints and numberOfValues:

grib_get -p numberOfPoints,numberOfValues,numberOfMissing surface.grib1
143 140 3

So that:

numberOfPoints = numberOfValues + numberOfMissing