You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 22 Current »

Contents of the sample static dataset

To help users begin planning for the migration, we have produced an initial static sample dataset which provides sample data encoded in GRIB edition 2 from: 

  • the analysis and forecast from the control (ex-HRES):
    • atmosphere:  stream=oper, type=an and type=fc
    • wave: stream=wave, type=an and type=fc
  • the perturbed members from the ENS forecast 
    • atmosphere stream=enfo, type=pf
    • wave:  stream=waef, type=pf

Sample data encoded in GRIB2 from the ensemble post-processed products (ensemble mean, maximum, etc, EFI and Sot), ensemble hindcasts, sub-seasonal and sub-seasonal hindcasts will also be made available in due course.

The sample data are based on the output from the current IFS cy49r1 output for the 00 UTC cycle on 30 November 2024 but encoded in GRIB2 as the data which will be produced by IFS cy51r1 are currently expected to be encoded.  All fields provided are global and interpolated to a 1.0° x1.0° regular latitude-longitude grid.

  • Data on constant pressure levels (type=pl) are provided at level=100 only.
  • Data on theta levels (type=pt) are provided at level=300 for type=fc and level=265 only for type=an.
  • Data on potential vorticity levels (levtype=pv) are provided at level=1500 only.
  • Data on model levels (levtype=ml) are provided at level=1 only.
  • Parameters accumulated from the start of the forecast have the ecCodes startStep key consistently set to 0 and, consequently, have stepRange=0-24, etc. 
  • Parameters accumulated from the start of the forecast are not provided at stepRange=0. 
  • Data from ENS forecasts are provided for a single ensemble member only (number=1).

Where to obtain the static sample dataset

  • Users with login access to the ECMWF HPC service can access the data from the directory /ec/vol/mtg2_sample. 
  • Users without login access to the ECMWF HPC or ECS service can download the data from https://data.ecmwf.int/mtg2_sample/.

On both the ECMWF HPC and the web server, the data are organised in a directory structure described in the next section.

Directory structure of static sample dataset

Data are ordered in a MARS-like folder structure under <root_dir>/<class>/<stream>/<type>/<levtype>.  Shown below is a diagrammatic layout of the structure under <root_dir>=/ec/vol/mtg2_sample/: 

<root_dir>/od/enfo
└── pf
    ├── pl
    ├── pt
    ├── pv
    ├── sfc
    └── sol

<root_dir>/od/oper
├── an
│   ├── ml
│   ├── pl
│   ├── pt
│   ├── pv
│   ├── sfc
│   └── sol
└── fc
    ├── ml
    ├── pl
    ├── pt
    ├── pv
    ├── sfc
    └── sol

<root_dir>/od/waef
└── pf
    └── sfc

<root_dir>/od/wave
├── an
│   └── sfc
└── fc
    └── sfc

Structure of file names

Each directory at the end of a branch of the tree contains one file per parameter with all forecast time steps.  The files are named <paramId>_<class>_<stream>_<type>_<levtype>_<shortName>_<year>_<expver>_<gridType>.grib2 and are based on the ecCodes paramId, shortName and gridType and the MARS class, stream, type, levtype used for the current operational output at IFS cy49r1.

For example, the file named 151_od_oper_fc_sfc_msl_2024_0001_regular_ll.grib2 in the <root_dir>/od/oper/fc/sfc/ directory contains the GRIB2 encoding of mean sea-level pressure from the Control (ex-HRES) forecast at all available forecast steps. 

Who should use the initial static sample dataset ?

The initial static sample dataset is aimed at GRIB encoding and decoding specialists that need early access to samples of ECMWF GRIB2 data to test and prepare any decoding software used that is not based on ECMWF ecCodes GRIB encoding and decoding software to handle the GRIB2 ECMWF data.  The dataset is not suitable for the general users of ECMWF data that want to test their full processing chain.

Sample data for the general user will be made available in the coming months when it will be possible to download the data from the ECMWF MARS archive and process the data using versions of ECMWF software, such as ecCodes and Metview that fully support the new ECMWF GRIB2 encoding.

What can be tested using the static sample dataset ?

The static sample dataset can be used for making some basic tests using your GRIB decoder to ensure the data can be read correctly.  In particular, testing is encouraged if you use a third-party GRIB decoder, such as wgrib2, which is not based on ecCodes.

If your decoder is based on ecCodes then ecCodes version 2.42.0 should be used  - see ecCodes version 2.42.0 released.  This is available for download from our ecCodes Releases page.

Note that ecCodes version 2.42.0 comes with a significant reworking of the concepts and lookup related to the paramIds/shortNames/names as well as the keys in the MARS namespace in preparation for the migration to GRIB2.  This mechanism is predominantly handled by the WMO tables version, with tablesVersion 33 imposing the pre or post migration behaviour.  All data using the MARS namespace will default to this switching behaviour, but some exceptions have been implemented to, for example, preserve ECMWF operations until IFS CY51R1.  See the ecCodes version 2.42.0 release notes for further details.

In particular, ecCodes version 2.42.0 should be used with care and not introduced into any operational processing chain unless it has been fully tested beforehand to ensure that none of the changes impact your production.

For users with access to the ECMWF Atos HPC, ecCodes 2.42.0 is also installed as part of the 2025.06.0.0 version of the ecmwf-toolbox.  This can be accessed on the Atos using:

$ module load ecmwf-toolbox/2025.06.0.0

$ grib_ls -V

ecCodes Version 2.42.0

Version 2025.06.0.0 of the ecmwf-toolbox also provides an installation of Metview linked with ecCodes 2.42.0 and which can be used to test Metview code that process the static sample data.

The static sample dataset can be decoded by versions of ecCodes older than 2.42.0 but some keys, in particular the shortName and paramId, may not be set correctly.  However, it is possible use the grib_dump tool to inspect the GRIB headers and here the keys are set to the correct values as far as we are aware.

In general, the static sample dataset  is not suitable for initialising an NWP model.  Although encoding examples of all parameters at all forecast steps are provided, for those parameters on constant pressure levels, for example, only the 1hPa level is provided.  However, users that use initial and lateral boundary conditions (LBCs) from ECMWF data to initialise and update their NWP models should note the change in the way the parameters on soil levels are provided as this may require changes to your code.  Instead of having four separate parameters at each soil layer (level), in GRIB2 there will be one parameter available at each of the four layers - see Soil temperature, volumetric soil moisture and sea-ice temperature, in Migration to GRIB2 - changes to encoding of parameters for details.

Other sources of ECMWF GRIB2 data that can be used for testing

Along with the static sample dataset, users can also use the ECMWF Open Data to test their processing of GRIB2 data.  All of the open data are encoded in GRIB2.  For more information about these data and how to obtain them, see ECMWF open data: real-time forecasts from IFS and AIFS.

Changes to encoding of parameter paramId, shortName, name and units

The encoding of the ecCodes paramId, shortName, name and units have changed for some surface parameters from the analysis and forecast.  Users may need to adapt their workflows in order to process these parameters successfully.  The main changes affecting parameters provided in the static sample dataset are summarised in Migration to GRIB2 - changes to encoding of parameters.  For a list of all changes to parameter encoding made at each ecCodes version, see Migration to GRIB2 - new in ecCodes.

  • No labels