Page History

...

Warning

title	Data checking

As for any good data archive it is crucial to have working data quality checking in place. Every data provider should double check that their parameters are fully compliant with required and agreed definitions linked in the main parameter page. A special attention before any archiving activities must be paid for example to correct units, fluxes sign conventions etc as specified in the GRIB2 encoding page. The additional checking tools provided by ECMWF and described below cannot discover all types of possible fundamental errors from principle (like incorrect units or reverse sign of fluxes where values range is both positive and negative).

...

UERRA-GRIB2 checking tool (tigge_check)
This tool should be run on all input files already in GRIB2 UERRA compliant format before archiving them. It checks all encoding details so that only fully compliant UERRA files following exactly required definitions would pass. It can check also allowed value ranges for each parameter if used with the option -v.

Warning

The tigge_check can check only the encoded GRIB2 keys to have them compliant with expected UERRA definitions specified generally for all UERRA datasets and each particular parameter. Some types of possible fundamental errors cannot be nevertheless revealed by the the tool from principle (e.g. incorrect units which are never encoded in GRIB2 files or a possibly wrong (reverse) sign of fluxes where values range is both positive and negative.

The data min/max value checking provided by tigge_check (-v) must be considered only as a helping option to have some chance to reveal sometimes real data issues. In some cases e.g. for radiation fluxes the allowed limits must be very flexible as for example direct solar radiation in 1-hourly outputs changes from 0 to 1+e^₉ depending on forecast step.

The tuning of the allowed limits for numerous parameters from different models on varying domains as in UERRA case is tricky and it is an ongoing process. On top of it there are sometimes clearly wrong values coming from some models during specific weather situations (e.g. grid point storms with wind speed exceeding 900 m/s). After an agreement with data provider the tool can allow such unrealistic values for given model as "normal". It should be understood that such data will have user impact and might be still considered as poor output data quality checking.

MARS archive content checking script
This kind of script is required to be run after each archiving to check that only expected fields were archived successfully (always the same parameters without any change). The checking script below is based on MARS list functionality.

Example of MARS list request checking the content of MARS for COSMO data from 1993-12-31.

Code Block

title	MARS checking scripts

list,
      class      = ur,
      stream     = oper,
      type       = all,
      date       = 19931231,
      time       = all,
      levtype    = all,
      origin     = eswi,
      expver     = test,
      hide       = file/length/offset/id/missing/cost/branch/date/hdate/month/year,
      target     = tree.out,
      database   = marsscratch,
      output     = tree

list,
      class      = ur,
      hide       = file/length/offset/id/missing/cost/branch/param/levtype/levelist/expver/type/class/stream/origin/date/time/step/number/hdate/month/year,
      target     = cost.out,
      output     = table

The tree.out content should be the same for all archived dates what can be easily checked e.g. with unix diff tool against the reference MARS list output created from the very 1st properly archived day for given model.

Code Block

title	tree.out content

class=ur,expver=test,levtype=hl,origin=eswi,stream=oper,type=an,time=00:00:00/06:00:00/12:00:00/18:00:00,param=10/130/157/3031/54,levelist=100/15/150/200/250/30/300/400/50/500/75
class=ur,expver=test,levtype=ml,origin=eswi,stream=oper,type=an,param=130/131/132/133,time=00:00:00/06:00:00/12:00:00/18:00:00,levelist=1/10/11/12/13/14/15/16/17/18/19/2/20/21/22/23/24/25/26/27/28/29/3/30/31/32/33/34/35/36/37/38/39/4/40/41/42/43/44/45/46/47/48/49/5/50/51/52/53/54/55/56/57/58/59/6/60/61/62/63/64/65/7/8/9
class=ur,expver=test,levtype=pl,origin=eswi,stream=oper,type=an,time=00:00:00/06:00:00/12:00:00/18:00:00,param=130/131/132/156/157,levelist=10/100/1000/150/20/200/250/30/300/400/50/500/600/70/700/750/800/825/850/875/900/925/950/975
class=ur,expver=test,levtype=sfc,origin=eswi,stream=oper,type=an,time=00:00:00/06:00:00/12:00:00/18:00:00,param=134/136/151/167/172/173/207/228002/228141/228164/235/260242/260260/260509/3073/3074/3075/33
class=ur,expver=test,levtype=sol,origin=eswi,stream=oper,type=an,param=260199/260360,levelist=1/2/3,time=00:00:00/06:00:00/12:00:00/18:00:00
class=ur,expver=test,levtype=hl,origin=eswi,stream=oper,type=fc,param=10/130/157/246/247/3031/54,levelist=100/15/150/200/250/30/300/400/50/500/75
 time=00:00:00/12:00:00,step=1/12/15/18/2/21/24/27/3/30/4/5/6/9
 time=06:00:00/18:00:00,step=1/2/3/4/5/6
class=ur,expver=test,levtype=pl,origin=eswi,stream=oper,type=fc,param=130/131/132/156/157/246/247/260257,levelist=10/100/1000/150/20/200/250/30/300/400/50/500/600/70/700/750/800/825/850/875/900/925/950/975
 time=00:00:00/12:00:00,step=1/12/15/18/2/21/24/27/3/30/4/5/6/9
 time=06:00:00/18:00:00,step=1/2/3/4/5/6
class=ur,expver=test,levtype=sfc,origin=eswi,stream=oper,type=fc
 time=00:00:00/06:00:00/12:00:00/18:00:00,step=1/2/3/4/5/6,param=134/136/146/147/151/167/169/173/174008/175/176/177/201/202/207/228141/228144/228164/228228/235/260242/260259/260260/260264/260430/260509/3073/3074/3075/33/49
 time=00:00:00/12:00:00,step=12/15/18/21/24/27/30/9,param=134/136/151/167/169/175/176/177/201/202/207/228144/228164/228228/235/260242/260259/260260/260264/3073/3074/3075/49
class=ur,expver=test,levtype=sol,origin=eswi,stream=oper,type=fc,param=260199/260360,levelist=1/2/3,time=00:00:00/06:00:00/12:00:00/18:00:00,step=1/2/3/4/5/6

Code Block

title	cost.out content

Grand Total:
============

Entries       : 13,852
Total         : 8,931,971,037 (8.31855 Gbytes)


> archived=$(cat cost.out| grep ^Entries|sed s/,//g| sed 's/.*: //')
> echo $archived
> 13852

The number of fields archived must be always the same. That number can be easily parsed from the above output for example using unix grep:

Code Block

title	Number of fields archived

> archived=$(cat cost.out| grep ^Entries|sed s/,//g| sed 's/.*: //')
> echo $archived
> 13852

Space shortcuts

Page tree

Versions Compared

Old Version 6

New Version Current

Key