SOFF initiative: technical documentation

How to access the data

With Linux FTP line command:

> ftp diss.ecmwf.int
# provide username (wmo_xx) and password

# list the available folders
> ls

# move into the folder for a specific day
> cd yyyymmdd

# download all the files in the folder
> mget * .

Filenaming convention and dissemination schedule

Once you access the FTP, you will see different folders in it, corresponding to the last 4 days of data (folder names in format dd-mm-yyyy 00:00.

If you click on one folder, you will find all the data disseminated for that day.

Files are disseminated according to Dissemination schedule.

For every folder, there are 4 different types of filenames, following this filenaming convention:

AADMMDDHHIImmddhhiiE where

AA can be one of the following values:
- AN (Analysis data)
- FM (Forecast model level)
- FP (Forecast pressure level)
- FS (Forecast surface level)
D means that the data come from ECMWF HRES data
MMDDHHII is month, day, hour and minute on which the products are based
mmddhhii is month, day, hour and minute on which the products are valid at (ii is set to 01 for high resolution forecast time step zero, type=fc, step=0)
E is the Experiment Version Number’ (as EXPVER in MARS, normally 1).

How to read the data - ecCodes Library

In order to open and read in your GRIB files, ECMWF has developed the ecCodes library, which is ECMWF's primary software package for decoding and encoding GRIB and BUFR messages. In ecCodes, keys are predefined names used to access specific pieces of metadata or data in a GRIB or BUFR message. Each key represents a distinct attribute or property of the data contained within a message.

`grib_ls`: Quick Summary of GRIB Messages

grib_ls allows users to quickly view key metadata from one or more GRIB messages. It prints a tabular summary that includes information such as the variable name, level, date and time.

Basic Usage

grib_ls 20250421000000-0h-oper-fc.grib2

Where 20250421000000-0h-oper-fc.grib2 is the name of the file you would like to summarise.

Sample Output

edition      centre       date         dataType     gridType     stepRange    typeOfLevel  level        shortName    packingType
2            ecmf         20250421     fc           regular_ll   0            surface      0            lsm          grid_ccsds

Some common Options that can be used include:

-p key1,key2,...: Display selected keys only.
-w key=value: Filter messages by key values.
-m: Displays the associated MARS keys.
-n namespace: Displays all the keys belonging to the namespace.

Use Case

You have downloaded a GRIB file containing surface-level forecast data. Before using the file for plotting or further processing, you want to check what parameters (e.g. 2 metre temperature, 10 metre wind U and V values, mean sea level pressure) and forecast steps are included in the file you have.

Solution: In this case, you can use the grib_ls command to get a quick overview of the file.

grib_ls -p shortName,levelType,level,stepRange,validityDate,validityTime 20250504000000-0h-oper-fc.grib2

`grib_dump`: Detailed Inspection of GRIB Content

grib_dump provides a full and detailed output of the contents of GRIB messages, including both metadata and data values. It is especially useful for debugging or exploring the internal structure of GRIB files.

Basic Usage

grib_dump 20250421000000-0h-oper-fc.grib2

Where 20250421000000-0h-oper-fc.grib2 is the name of the file you would like to inspect.

Sample Output (One Message Snippet)

#==============   MESSAGE 160 ( length=723061 )            ==============
GRIB {
  # Meteorological products (grib2/tables/34/0.0.table)
  discipline = 0;
  editionNumber = 2;
  # European Centre for Medium-Range Weather Forecasts (common/c-11.table)
  centre = 98;
  subCentre = 0;
  # Start of forecast (grib2/tables/34/1.2.table)
  significanceOfReferenceTime = 1;
  dataDate = 20250421;
  dataTime = 0;
  # Operational products (grib2/tables/34/1.3.table)
  productionStatusOfProcessedData = 0;
  # Forecast products (grib2/tables/34/1.4.table)
  typeOfProcessedData = 1;
  # MARS labelling (grib2/grib2LocalSectionNumber.98.table)
  grib2LocalSectionNumber = 1;
  # Operational archive (mars/class.table)
  marsClass = 1;
  # Forecast (mars/type.table)
  marsType = 9;
  # Atmospheric model (mars/stream.table)
  marsStream = 1025;
  experimentVersionNumber = 0001;
  numberOfDataPoints = 1038240;
  # There is no appended list (grib2/tables/34/3.11.table)
  interpretationOfNumberOfPoints = 0;
  # Latitude/longitude (Also called equidistant cylindrical, or Plate Carree)  (grib2/tables/34/3.1.table)
  gridDefinitionTemplateNumber = 0;
  # Earth assumed spherical with radius of 6 371 229.0 m (grib2/tables/34/3.2.table)
  shapeOfTheEarth = 6;
  Ni = 1440;
  Nj = 721;
  iScansNegatively = 0;
  jScansPositively = 0;
  jPointsAreConsecutive = 0;
  alternativeRowScanning = 0;
  latitudeOfFirstGridPointInDegrees = 90;
  longitudeOfFirstGridPointInDegrees = 180;
  latitudeOfLastGridPointInDegrees = -90;
  longitudeOfLastGridPointInDegrees = 179.75;
  iDirectionIncrementInDegrees = 0.25;
  jDirectionIncrementInDegrees = 0.25;
  gridType = regular_ll;
  NV = 0;
  # Analysis or forecast at a horizontal level or in a horizontal layer at a point in time (grib2/tables/34/4.0.table)
  productDefinitionTemplateNumber = 0;
  # Temperature (grib2/tables/34/4.1.0.table)
  parameterCategory = 0;
  # Temperature (K)  (grib2/tables/34/4.2.0.0.table)
  parameterNumber = 0;
  #-READ ONLY- parameterUnits = K;
  #-READ ONLY- parameterName = Temperature;
  # Forecast (grib2/tables/34/4.3.table)
  typeOfGeneratingProcess = 2;
  generatingProcessIdentifier = 158;
  # Hour (grib2/tables/34/4.4.table)
  indicatorOfUnitForForecastTime = 1;
  stepUnits = h;
  forecastTime = 0;
  stepRange = 0;
  # Isobaric surface (Pa)  (grib2/tables/34/4.5.table)
  typeOfFirstFixedSurface = 100;
  #-READ ONLY- unitsOfFirstFixedSurface = Pa;
  #-READ ONLY- nameOfFirstFixedSurface = Isobaric surface;
  scaleFactorOfFirstFixedSurface = 0;
  scaledValueOfFirstFixedSurface = 15000;
  # Missing (grib2/tables/34/4.5.table)
  typeOfSecondFixedSurface = 255;
  #-READ ONLY- unitsOfSecondFixedSurface = unknown;
  #-READ ONLY- nameOfSecondFixedSurface = Missing;
  scaleFactorOfSecondFixedSurface = MISSING;
  scaledValueOfSecondFixedSurface = MISSING;
  level = 150;
  shortName = t;
  name = Temperature;
  cfName = air_temperature;
  #-READ ONLY- cfVarName = t;
  modelName = IFS;
  #-READ ONLY- modelVersion = cy49r1;
  numberOfValues = 1038240;
  packingType = grid_ccsds;
  ccsdsFlags = 14;
  ccsdsBlockSize = 32;
  ccsdsRsi = 128;
  # A bit map does not apply to this product (grib2/tables/34/6.0.table)
  bitMapIndicator = 255;
  bitmapPresent = 0;
  values(1038240) =  {
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241,
  226.241, 226.241, 226.241, 226.241, 226.241
  ... 1038140 more values
  }
  #-READ ONLY- maximum = 230.554;
  #-READ ONLY- minimum = 198.749;
  #-READ ONLY- average = 215.112;
  #-READ ONLY- standardDeviation = 7.31015;
  #-READ ONLY- skewness = 0.149981;
  #-READ ONLY- kurtosis = -1.23212;
  #-READ ONLY- isConstant = 0;
  #-READ ONLY- numberOfMissing = 0;
  #-READ ONLY- getNumberOfValues = 1038240;
}

Some common Options that can be used include:

-a: If the file contains multiple messages, it will dump all of them (this is the default behaviour, but you can use -a for dumping aliases).
-p key1,key2,...: If you want to dump specific keys from each message.
-O: enables Octet mode, producing a WMO documentation-style dump of the GRIB messages.
-d: prints all data values in the GRIB messages.

`grib_to_netcdf`: Convert GRIB to NetCDF

ecCodes also allows you to convert your GRIB data into NetCDF files using the grib_to_netcdf command.

Basic Usage

grib_to_netcdf -o 20250421000000-0h-oper-fc_converted.nc 20250421000000-0h-oper-fc.grib2

Where -o specifies the name of the NetCDF output file and 20250421000000-0h-oper-fc.grib2_converted.nc is whatever name you specify for the output file.

Sample Output

grib_to_netcdf: Version 2.40.0
grib_to_netcdf: Processing input file '20250421000000-0h-oper-fc.grib2'.
grib_to_netcdf: Found 160 GRIB fields in 1 file.
grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
grib_to_netcdf: Creating netCDF file '20250421000000-0h-oper-fc_converted.nc'

Some common Options that can be used include:

-I key1,key2,...: ignores specific keys during conversion. By default: method, type, stream, refdate, expver are ignored.
-S key1,key2,...: splits the GRIB file according to specified keys. By default it will split: param, expver.
-k kind: specifies the type of NetCDF file to create. Possible values are:
- 1 -> netCDF classic file format
  2 -> netCDF 64 bit classic file format (Default)
  3 -> netCDF-4 file format
  4 -> netCDF-4 classic model file format

`grib_copy:` Copy Contents of a GRIB File

grib_copy copies the content of GRIB files, printing values of some keys.

Basic Usage

grib_copy 20250421000000-0h-oper-fc.grib2 output.grib2

Where 20250421000000-0h-oper-fc.grib2 is your input file to copy and output.grib2 is whatever name you specify for the output file. Note here that using grib_copy without options will take GRIB files with multi-field messages and convert them into single-field messages.

Some common Options that can be used include:

-w key[:{s|d|i}]{=|!=}value,key[:{s|d|i}]=value,...: a 'Where' clause. Only GRIB messages matching the key/value constraints are copied to the output file. A valid constraint is of type key=value or key!=value. For each key a string (key:s), a double (key:d) or an integer (key:i) type can be defined. Default type is string. In the value, you can also use the forward-slash character '/' to specify an OR condition (i.e. a logical disjunction). Note: only one -w clause is allowed.
-p key[:{s|d|i}],key[:{s|d|i}],..: declaration of keys to print. For each key a string (key:s), a double (key:d) or an integer (key:i) type can be requested. Default type is string.

Processing data - recipes

In order to process ECMWF Model level data, additional steps are required but there are recipes already available on-line e.g.:

By converting parameters on model levels to GRIB1
https://medium.com/@valcap74/how-to-run-wrf-model-driven-by-era5-on-model-levels-3c3e31838d22
https://confluence.ecmwf.int/display/OIFSUF/Grib+to+Netcdf+conversion
Directly from GRIB2 with an external Vtable exclusively from ML in GRIB2: https://earthscience.stackexchange.com/questions/9497/running-wrf-with-ecmwf-grib2-model-levels
Since wgrib has separate executables for GRIB1 and GRIB2, wgrib and therefore WPS cannot process mixed files containing GRIB1 and GRIB2. Therefore it is required to have ML data (GRIB2) and SFC data (GRIB1) is separate files and to un ungrib independently for GRIB1 and again for GRIB2 before running metgrid. It is required to make following changes in namelist.wps between two ungrib runs:

&ungrib
 prefix = 'SFILE' 
&ungrib
  prefix = 'MFILE'

To ensure that the two ungrib runs do not overwrite each other.
Then in metgrid section of namelist.wps you need to make sure to use both files created by ungrib runs:

 &metgrid
 fg_name = 'SFILE' 'MFILE'

What has changed since these recipes are written is that now some GRIB2 parameters are compressed using CCSDS compression which WRF Pre-Processing System cannot decode. Consequently, CCSDS packing in GRIB2 has to be changed to SIMPLE PACKING before any of these steps with:

grib_set -r -w packingType=grid_ccsds -s packingType=grid_simple ccsds.grib grid_simple.grib

Product description

The products are High-Resolution (HRES) forecast and analysis data with a 0.1° by 0.1° grid (a subset of ECMWF Set-I).

Parameters

Surface level data - Analysis (ANDMMDDHHIImmddhhiiE files)
Analysis fields can be provided for base time 00, 06, 12 or 18

Short name	Long name	Unit	ID	Level	Steps for times 00 & 12	Steps for time 06 & 18	Domain	Access
slt	Soil type	~	43	Surface	0	0	Dependent on region

Surface level data - Forecast (FSDMMDDHHIImmddhhiiE files)

Short name	Long name	Unit	ID	Level	Steps for times 00 & 12	Steps for time 06 & 18	Domain
slt	Soil type	~	43	Surface	0	0	Dependent on region
10u	10 metre U wind component	m s**-1	165	Surface	0 to 90 by 3	-	Dependent on region
10v	10 metre V wind component	m s**-1	166	Surface	0 to 90 by 3	-	Dependent on region
2t	2 metre temperature	K	167	Surface	0 to 90 by 3	-	Dependent on region
2d	2 metre dewpoint temperature	K	168	Surface	0 to 90 by 3	-	Dependent on region
lsm	Land-sea mask	(0 - 1)	172	Surface	0 to 90 by 3	-	Dependent on region
sp	Surface pressure	Pa	134	Surface	0 to 90 by 3	-	Dependent on region
msl	Mean sea level pressure	Pa	151	Surface	0 to 90 by 3	-	Dependent on region
skt	Skin temperature	K	235	Surface	0 to 90 by 3	-	Dependent on region
src	Skin reservoir content	m of water equivalent	198	Surface	0 to 90 by 3	-	Dependent on region
ci	Sea ice area fraction	(0 - 1)	31	Surface	0 to 90 by 3	-	Dependent on region
sst	Sea surface temperature	K	34	Surface	0 to 90 by 3	-	Dependent on region
rsn	Snow density	kg m**-3	33	Surface	0 to 90 by 3	-	Dependent on region
sd	Snow depth	m of water equivalent	141	Surface	0 to 90 by 3	-	Dependent on region
asn	Snow albedo	(0 - 1)	32	Surface	0 to 90 by 3	-	Dependent on region
tsn	Temperature of snow layer	K	238	Surface	0 to 90 by 3	-	Dependent on region
stl1	Soil temperature level 1	K	139	Surface	0 to 90 by 3	-	Dependent on region
stl2	Soil temperature level 2	K	170	Surface	0 to 90 by 3	-	Dependent on region
stl3	Soil temperature level 3	K	183	Surface	0 to 90 by 3	-	Dependent on region
stl4	Soil temperature level 4	K	236	Surface	0 to 90 by 3	-	Dependent on region
swvl1	Volumetric soil water layer 1	m3 m-3	39	Surface	0 to 90 by 3	-	Dependent on region
swvl2	Volumetric soil water layer 2	m3 m-3	40	Surface	0 to 90 by 3	-	Dependent on region
swvl3	Volumetric soil water layer 3	m3 m-3	41	Surface	0 to 90 by 3	-	Dependent on region
swvl4	Volumetric soil water layer 4	m3 m-3	42	Surface	0 to 90 by 3	-	Dependent on region
z	Geopotential	m2 s-2	129	Surface	0 to 90 by 3	-	Dependent on region

Pressure level data - Forecast (FPDMMDDHHIImmddhhiiE files)

Short name	Long name	Unit	ID	Level	Steps for times 00 & 12	Steps for time 06 & 18	Domain
T	Temperature	K	130	100, 200, 250, 500, 850, 1000	0 to 90 by 3	-	Dependent on region
U	U component of wind	m s**-1	131	100, 200, 250, 500, 850, 1000	0 to 90 by 3	-	Dependent on region
V	V component of wind	m s**-1	132	100, 200, 250, 500, 850, 1000	0 to 90 by 3	-	Dependent on region
R	Relative humidity	%	157	100, 200, 250, 500, 850, 1000	0 to 90 by 3	-	Dependent on region
Z	Geopotential	m2 s-2	129	100, 200, 250, 500, 850, 1000	0 to 90 by 3	-	Dependent on region

Model level data - Forecast (FMDMMDDHHIImmddhhiiE files)

Short name	Long name	Unit	ID	Level	Steps for times 00 & 12	Steps for time 06 & 18	Domain
T	Temperature	K	130	1 to 137	0 to 90 by 3	-	Dependent on region
U	U component of wind	m s**-1	131	1 to 137	0 to 90 by 3	-	Dependent on region
V	V component of wind	m s**-1	132	1 to 137	0 to 90 by 3	-	Dependent on region
W	Vertical velocity	m s**-1	135	1 to 137	0 to 90 by 3	-	Dependent on region
Q	Specific humidity	kg kg**-1	133	1 to 137	0 to 90 by 3	-	Dependent on region
CLWC	Specific cloud liquid water content	kg kg**-1	246	1 to 137	0 to 90 by 3	-	Dependent on region
CIWC	Specific cloud ice water content	kg kg**-1	247	1 to 137	0 to 90 by 3	-	Dependent on region
CRWC	Specific rain water content	kg kg**-1	75	1 to 137	0 to 90 by 3	-	Dependent on region
LNSP	Logarithm of surface pressure	~	152	1	0 to 90 by 3	-	Dependent on region

Type:
Real-time

Page tree

SOFF initiative: technical documentation

How to access the data

Filenaming convention and dissemination schedule

How to read the data - ecCodes Library

grib_ls: Quick Summary of GRIB Messages

Basic Usage

Sample Output

Use Case

grib_dump: Detailed Inspection of GRIB Content

Basic Usage

Sample Output (One Message Snippet)

grib_to_netcdf: Convert GRIB to NetCDF

Basic Usage

Sample Output

grib_copy: Copy Contents of a GRIB File

Basic Usage

Processing data - recipes

Product description

`grib_ls`: Quick Summary of GRIB Messages

`grib_dump`: Detailed Inspection of GRIB Content

`grib_to_netcdf`: Convert GRIB to NetCDF

`grib_copy:` Copy Contents of a GRIB File