You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

  • The archiving of UERRA data in MARS should be done by each provider on his own with full support of ECMWF's team.
  • The ECMW's account is needed to be able to archive data in MARS and access the database online or via web-api interface.

3 steps of test to production archiving

  1. MARSSCRATCH:
    The MARSSCRATCH is temporary development version of MARS database. Anybody can use it for the first preliminary tests but the data can disappear at any time even it is not happening most of the time.
    Example of COSMO sample data stored in MARSSCRATCH: http://apps.ecmwf.int/mars-servers/marsscratch/?origin=edzw&stream=oper&class=ur&expver=test
  2. MARS (expver=test)
    Before the full production archiving (step 3) a few days of full UERRA data from given model are required to be archived in MARS with expver=test. It must be thoroughly checked that all fields are archived as expected (no missing or additional ones; no missing or additional steps, levels etc.). The data archived as expver=test might be overwritten during this phase.
  3. MARS (expver=prod)
    Once all providers' processing and archiving scripts are verified as fully working during step 2, the production archiving can start. The production data in MARS might not be overwritten in the future so the step 2 has crucial importance. It is also very difficult to delete any unwanted data from production MARS e.g. archived by mistake so it should be avoided by all means. Only one user ID per provider is expected to have granted permissions for archiving in MARS to avoid any kind of possible double archiving in the same time etc.

Effective archiving

It is important to archive UERRA data effectively following MARS design specific for this type of dataset. The possible suboptimal archiving (e.g. one parameter by one in each archiving request instead of archiving them all together) would cause various MARS performance issues which could be avoided.

In the archiving scripts which will be provided to each partner (see below)  it will be archived always one day at once for given origin, stream and levtype (i.e. all parameters, times, steps and levels together). If any MARS performance issue occurs during test and production archiving with that approach it might be needed to change it.

As a consequence of MARS design for UERRA datasets up to one full month could be theoretically archived at once (again for given origin, stream and levtype as usually).

Tools

Following tools will be available for providers to allow smooth data processing and archiving in MARS:

Conversion script from GRIB1 to UERRA compliant GRIB2

For models which produce the input data for MARS still in GRIB1 the conversion scripts to the required GRIB2 format will be provided. They are based on GRIB-API grib_filter tool which requires the GRIB-API version fully supporting all UERRA parameters.

Examples of grib_filter rules for HARMONIE parameters in GRIB1 conversion:

  • rules.hirlam.convert.sl.batch
  • rules.hirlam.convert.vl.batch

Archiving request

An example how to archive full data sample produced by given model will be provided. Basically only the date should change in the script otherwise the content of archive is expected not to be changing for the whole archiving period (i.e. homogeneous data without gaps or any change is expected).

Example of MARS archiving request for full surface, soil and model level data from HARMONIE deterministic reanalysis (origin=eswi, stream=oper, type=an):

MARS archiving request
archive,class=ur,database=marsscratch,
stream=oper,type=an,levtype=sfc,expver=TEST,date=$date,origin=eswi,time=0/6/12/18,
param=lcc/lsm/msl/skt/sp/orog/mcc/rsn/sd/hcc/10wdir/2t/al/10si/sr/tcw/2r/tcc,
level=0/2/10,
step=0,
number=off,
expect=72,
source=an.$date.sfc.grib2

archive,class=ur,database=marsscratch,
stream=oper,type=an,levtype=sol,expver=TEST,date=$date,origin=eswi,time=0/6/12/18,
param=vsw/sot,
level=1/to/3,
step=0,
number=off,
expect=24,
source=an.$date.sol.grib2

archive,class=ur,database=marsscratch,
stream=oper,type=an,levtype=ml,expver=TEST,date=$date,origin=eswi,time=0/6/12/18,
param=u/t/v/q,
level=1/to/63,
step=0,
number=off,
expect=1040,
source=an.$date.ml.grib2

Checking tools

There are 2 types of checking tools which must be run before and after archiving to try to minimize possible errors in the archive

  1. UERRA-GRIB2 checking tool (tigge_check)
    This tool should be run on all input files before archiving them. It is checking all encoding details so that only fully compliant UERRA files following exactly required definitions would pass. It can check also allowed value ranges for each parameter if used with the option -v.

  2. MARS archive content checking script
    This kind of script is required to be run after each archiving to check that the expected fields were archived successfully (always the same parameters without any change). The checking script below is based on MARS list functionality.

    Example of MARS list request checking the content of MARS for COSMO data from 1993-12-31.

    MARS checking scripts
    list,
      class      = ur,
      stream     = oper,
      type       = all,
      date       = 19931231,
      time       = 00/06/12/18,
      levtype    = ml/pl/hl/sfc/sol,
      origin     = eswi,
      expver     = test,
      hide       = file/length/offset/id/missing/cost/branch/date/hdate/month/year,
      target     = tree.out,
      database   = marsscratch,
      output     = tree

list,
      class      = ur,
      hide       = file/length/offset/id/missing/cost/branch/param/levtype/levelist/expver/type/class/stream/origin/date/time/step/number/hdate/month/year,
      target     = cost.out,
      output     = table

    The tree.out content should be the same for all dates what can be easily checked e.g. with unix diff tool against the reference MARS list output created from the very 1st properly archived day for given model.

    tree.out content
    class=ur,expver=test,levtype=hl,origin=eswi,stream=oper,type=an,time=00:00:00/06:00:00/12:00:00/18:00:00,param=10/130/157/3031/54,levelist=100/15/150/200/250/30/300/400/50/500/75
class=ur,expver=test,levtype=ml,origin=eswi,stream=oper,type=an,param=130/131/132/133,time=00:00:00/06:00:00/12:00:00/18:00:00,levelist=1/10/11/12/13/14/15/16/17/18/19/2/20/21/22/23/24/25/26/27/28/29/3/30/31/32/33/34/35/36/37/38/39/4/40/41/42/43/44/45/46/47/48/49/5/50/51/52/53/54/55/56/57/58/59/6/60/61/62/63/64/65/7/8/9
class=ur,expver=test,levtype=pl,origin=eswi,stream=oper,type=an,time=00:00:00/06:00:00/12:00:00/18:00:00,param=130/131/132/156/157,levelist=10/100/1000/150/20/200/250/30/300/400/50/500/600/70/700/750/800/825/850/875/900/925/950/975
class=ur,expver=test,levtype=sfc,origin=eswi,stream=oper,type=an,time=00:00:00/06:00:00/12:00:00/18:00:00,param=134/136/151/167/172/173/207/228002/228141/228164/235/260242/260260/260509/3073/3074/3075/33
class=ur,expver=test,levtype=sol,origin=eswi,stream=oper,type=an,param=260199/260360,levelist=1/2/3,time=00:00:00/06:00:00/12:00:00/18:00:00
class=ur,expver=test,levtype=hl,origin=eswi,stream=oper,type=fc,param=10/130/157/246/247/3031/54,levelist=100/15/150/200/250/30/300/400/50/500/75
 time=00:00:00/12:00:00,step=1/12/15/18/2/21/24/27/3/30/4/5/6/9
 time=06:00:00/18:00:00,step=1/2/3/4/5/6
class=ur,expver=test,levtype=pl,origin=eswi,stream=oper,type=fc,param=130/131/132/156/157/246/247/260257,levelist=10/100/1000/150/20/200/250/30/300/400/50/500/600/70/700/750/800/825/850/875/900/925/950/975
 time=00:00:00/12:00:00,step=1/12/15/18/2/21/24/27/3/30/4/5/6/9
 time=06:00:00/18:00:00,step=1/2/3/4/5/6
class=ur,expver=test,levtype=sfc,origin=eswi,stream=oper,type=fc
 time=00:00:00/06:00:00/12:00:00/18:00:00,step=1/2/3/4/5/6,param=134/136/146/147/151/167/169/173/174008/175/176/177/201/202/207/228141/228144/228164/228228/235/260242/260259/260260/260264/260430/260509/3073/3074/3075/33/49
 time=00:00:00/12:00:00,step=12/15/18/21/24/27/30/9,param=134/136/151/167/169/175/176/177/201/202/207/228144/228164/228228/235/260242/260259/260260/260264/3073/3074/3075/49
class=ur,expver=test,levtype=sol,origin=eswi,stream=oper,type=fc,param=260199/260360,levelist=1/2/3,time=00:00:00/06:00:00/12:00:00/18:00:00,step=1/2/3/4/5/6
    Content of cost.out
    Grand Total:
============

Entries       : 13,852
Total         : 8,931,971,037 (8.31855 Gbytes)


> archived=$(cat cost.out| grep ^Entries|sed s/,//g| sed 's/.*: //')
> echo $archived
> 13852

    The number of fields archived must be always the same. That reference number can be easily parsed from the above output for example using unix grep:

    Number of fields archived
    > archived=$(cat cost.out| grep ^Entries|sed s/,//g| sed 's/.*: //')
> echo $archived
> 13852

     

     

  • No labels