Skip to end of metadata
Go to start of metadata


Production of the  data files with the new Product Generation software is fully supported operationally at ECMWF. Please report any operational issues to servicedesk@ecmwf.int.

Table of contents

Background

Product Generation is the process by which direct model output is converted into user's data files according to their requirements. The interpolation library lies at its core.

The Product Generation software has undergone a complete rewrite to include the new interpolation package MIR and introduce a number of other efficiency improvements.

News

See announcements here

Important dates

  • 9 April 2019: Product Generation with EMOSLIB decommissioned.
  • 14th of January 2019: Feedback with the results of the testing phase to be sent to Data.Services
  • End of November 2018: Product Generation with MIR is introduced into operations in parallel with the present Product Generation with EMOSLIB.

Timeline

Product Generation with MIR will be released by the end of November, giving users the opportunity to migrate as soon  as they are ready after the release.
Product Generation with EMOSLIB will run in parallel with Product Generation with MIR for a period of three months.  After this period,  Product Generation with EMOSLIB will be decommissioned.



How to test

Users that do not have access to ECPDS

Please send an email to Data.Services@ecmwf.int to request the test files to be delivered to your systems.

Users that have access to ECPDS

You will be able to access test data files at:

https://ecpds-xmonitor.ecmwf.int/do/start


Please ensure that firewalls accept internet transfers from the following servers:

193.61.196.104 (http://ecpds-xma.ecmwf.int)

193.61.196.105 (http://ecpds-xmb.ecmwf.int)

193.61.196.124


Login with your ECMWF user ID and the password obtained from the security token.
 
Test files have an additional extension .pgen so they won't overwrite your operational files. If you download them, they should be delivered to the same directory/host as your operational files.

When the new product generation software goes into production, files will no longer have the ".pgen" suffix. Your current file names will remain.

Please check the configuration of the host and directory configured on the test ECPDS as this may have been configured differently in the past (e.g. if you have requested a different configuration to test previous e-suites).

If you wish to change the receiving host/directory for the test files, please send an email to: Data.Services@ecmwf.int

Differences on the data files


Do not rely on the order of the fields in the file.  Files produced by Product Generation can contain fields in any order which can change from day to day.

This section is a compilation of the known differences presented by the output data when generated by the present Product Generation with EMOSLIB (PRODGEN) versus the new Product Generation with MIR  (PGEN).

It does not explain the differences between the two software.

Users  requesting reduced Gaussian grids will need to update the version of ecCodes  to 2.9.2  (or later)

Different interpolation library

Different data values

PGEN software uses MIR interpolation library and therefore data values will not be exactly the same. Prodgen uses EMOSLIB which is no longer supported. These differences are similar to, but generally much smaller than, those arising from a change of the forecast model.

See MARS interpolation with MIR for further information about differences  between EMOSLIB and MIR. The section  Highlights and main differences plots a number of illustrative fields showing the differences in the data values.

Different point count

Specific cases for  reduced Gaussian grids present different point count.  Any users affected by this have already been informed.

Differences in the parameter Relative Humidity (paramId=157)

The way of generating the Relative Humidity product has changed in PGEN. Prodgen using EMOSLIB was post-processing this product to limit the values between 0 and 100. Now we took the decision not to post-process this parameter any more so the values will be more similar to what the model produces. The values will now be as close as possible to what MARS interpolation with MIR produces.

Note that now you may find values over 100 (supersaturation) and small values <0. See IFS documentation for relative humidity.

Differences for area encoding

Different longitude encoding in GRIB1

PGEN is encoding the west/east longitudes differently  in GRIB1. 

  • prodgen will encode longitude -180 as 180, pgen will encode -180
  • prodgen will encode longitude 359 as -1, pgen will encode it as 359


User inputPRODGENPGEN

area=90/-180/-90/179.5

grid=0.5/0.5

area=90/180/-90/179.5


area=90/-180/-90/179.5


area=90/0/-90/359

grid=1/1

area=90/0/-90/-1


area=90/0/-90/359


area=90/-60/25/60

grid=640

gaussian=reduced

area=89.892/300/25.092/60


area=89.892/-60/25.092/60


Different precision 

In GRIB1, the precision is millidegrees; in GRIB2, it is microdegrees.  PGEN lets ecCodes decide on the best precision. PRODGEN encodes the areas in millidegrees, therefore the encoded values in GRIB2 can be different.

This case will affect mostly Gaussian grids:

User inputPRODGENPGEN

area=90/-60/25/60

grid=640

gaussian=regular

form=GRIB2

area=89.892/300.093/25.092/59.906


area=89.8924/300.094/25.0918/59.9062



Different rounding

User inputPRODGENPGEN

area=90/-60/25/60

grid=640

gaussian=regular

area=89.892/-59.907/25.092/59.906


area=89.892/-59.906/25.092/59.906



Different east longitude

When Gaussian grids are requested, PRODGEN sets the east latitude in the GRIB header to the user provided value (modulo 360 and modulo the GRIB1 precision) PGEN will put the longitude of the most eastern point.

User inputPRODGENPGEN

area=90/0/-90/359.9

grid=128

gaussian=reduced

area=89.463/0/-89.463/359.9

gridname=N128

area=89.463/0/-89.463/359.297

gridname=N128

Constant fields

Fields that may contain constant values are  packed by PGEN with bitsPerValue=0. This is a more efficient way to encode such fields and leads to smaller files.

An example of such field is CLWC on model levels high in the atmosphere where the values are zero at all grid points.

As a result, file sizes may change from one day to the next. 

PRODGEN encodes these with a bitsPerValue=8 so file sizes are the same from one day to the next.

Local definitions

PRODGEN does not produce files in GRIB1 with local definitions for 00 and 12 UTC HRES fields.  They are kept for 06 and 18 BC and ENS (including ENS-extended and SEAS).

The local definitions are (currently) retained by PGEN.

E.g given two files:

H1D09060000091600001.pgen

H1D09060000091600001

Please retain the order of the files when doing the comparison

grib_compare -H H1D09060000091600001.pgen H1D09060000091600001                                      

-- GRIB #1 -- shortName=sd paramId=141 stepRange=240 levelType=sfc level=0 packingType= gridType= --
long [totalLength]: [26534] != [26510]
long [section1Length]: [52] != [28]
[reservedNeedNotBePresent] not found in 2nd field
[localDefinitionNumber] not found in 2nd field
[marsClass] not found in 2nd field
[marsType] not found in 2nd field
[marsStream] not found in 2nd field
[experimentVersionNumber] not found in 2nd field
[perturbationNumber] not found in 2nd field
[numberOfForecastsInEnsemble] not found in 2nd field
[padding_local1_1] not found in 2nd field

Duplicated fields

PGEN does not deliver duplicated fields  configured in the requirements.   e.g. In the request below the parameter 2T will be delivered twice by PRODGEN and only once by PGEN

DIS,
TARGET 	=	ECM:EC,
STREAM 	= 	DA,
TYPE 	= 	FC,
GRID 	= 	0.125/0.125,
AREA 	= 	55/2.5/47/15.5,
TIME 	= 	00,
LEVTYPE 	= 	SFC,
STEP 	= 	0/to/168/by/3,
PARAM 	= 	2T/SSRD/2T

We have cleaned all requirements to remove any instance where a parameter is duplicated.

  • No labels