Description
For the projects TIGGE, S2S and UERRA the exact data format, WMO compliant GRIB2, is required to allow easy data processing and intercomparison. Read in Data encoding checking tools more information about tigge_check and grib_check.py tools. Apart from encoding verification, theses tools can do also some basic quality control by checking the allowed value ranges, if they were defined, for each parameter.
Please be aware that some previously allowed valid ranges can become obsolete at some point e.g. due to model upgrade to higher resolution meaning in general different values of some parameters. Because of that a continuous set up updates are needed so it is important to use the latest versions of the tools. Also for some parameters the allowed ranges must not be defined due to various reasons. In some cases the allowed limits for different data sets or data origins might also exist because of known special features or deficiencies of some related model's outputs.
Examples of tigge_check usage
Checking TIGGE data
module load eccodes load eccodes 2.9.2 Library (ECCODES_DIR, ECCODES_VERSION, ECCODES_INCLUDE, ECCODES_LIB, ECCODES_INCLUDE_DIR, ECCODES_LIB_DIR) # example with one data value ranges warning tigge_check -u -w -v tigge_mogreps-g_2019021600_00.grib2 warning: tigge_mogreps-g_2019021600_00.grib2, field 513 (Potential vorticity, paramId=60), matched definition (class:ti, model:glob, origin:egrr) => maximum value 1296779.249971 is not in [0.000001,100.000000] |
Examples of grib_check.py usage
grib_check options
Code Block |
---|
usage: grib_check.py [-h] [-v VERBOSITY] [-w] [-f] [-his] [-l [LISTING]]
[inp_file [inp_file ...]]
positional arguments:
inp_file enter input file name(s)
optional arguments:
-h, --help show this help message and exit
-v VERBOSITY, --verbosity VERBOSITY
increase output verbosity [0-2]
-w, --warning warnings are treated as errors..
-f, --fail force abort for undefined parameters..
-his, --history add value ranges history, if exists, to listing..
-l [LISTING], --listing [LISTING]
list defined parameters (search if argument provided!)
|
Checking LC-WVF data
export BIN=/home/ma/emos/def/s2s/s2s_prod/bin
python $BIN/grib_check.py lw.grib2 warning: lw.grib2, field 700 (Peak wave period, paramId=140231), matched definition (default) => minimum value -9.699028 is not in [0.000000,35.000000] warning: lw.grib2, field 718 (Peak wave period, paramId=140231), matched definition (default) => maximum value 41.463626 is not in [0.000000,35.000000] |
Checking parameter set up
List all defined limits
python $BIN/grib_check.py -l 165 [10 metre U wind component] [min <-150, -0.1> max <0.1, 150>] (default) [min <-150, -0.1> max <0.1, 150>] (class:ti, model:lam, suiteName:mogreps-mo-eua) 166 [10 metre V wind component] [min <-100, -1> max <0.5, 100>] (default) [min <-100, 10> max <-10, 100>] (class:ti, model:lam, suiteName:mogreps-mo-eua) 260260 [10 metre wind direction] [min <0, 1> max <359, 360.1>] (default)
...
|
See allowed limits for parameters with paramId containing 141
python $BIN/grib_check.py -l 141 228141 [Snow depth water equivalent] [min <-1e-10, 0> max <100, 15000>] (default) [min <-1e-10, 0> max <100, 40000>] (class:s2, origin:cwao) [min <-1, 0> max <100, 100000>] (class:ti) |
See allowed limits for parameters with paramId containing 141 including past set up history
python $BIN/grib_check.py -l 141 -his 228141 [Snow depth water equivalent] [min <-1e-10, 0> max <100, 15000>] (default)
# History of changes: # # 1.version (valid from: 2017-01-01) # min <-1e-10, 0> # max <100, 15000> # info:See long history of changes in tigge_check.h!
228141 [Snow depth water equivalent] [min <-1e-10, 0> max <100, 40000>] (class:s2, origin:cwao)
# History of changes: # # 1.version (valid from: 2013-02-11) # min <-1e-10, 0> # max <100, 40000> # info:s2s.z_s2s_c_cwao_20130211000000_glob_prod_sl_000.grib2, field 23: snow_depth_water_equivalent_sfc.glob.s2 maximum value 30452.5 is not in [100,15000]
228141 [Snow depth water equivalent] [min <-1, 0> max <100, 100000>] (class:ti)
# History of changes: # # 3.version (valid from: 2017-08-08) # min <-1, 0> # max <100, 100000> # info:z_tigge_c_RKSL_20170808000000_glob_prod_cf_0240_000.grib2, field 119 (Snow depth water equivalent, paramId=228141), matched definition (default) => maximum value 94537.8 is not in [100,15000] # 2.version (valid from: 2017-08-08) # min <-1, 0> # max <100, 90000> # info:warning: tigge_mogreps-g_2017080812_00.grib2, field 445 => maximum value 67640.3 is not in [100,15000] # 1.version (valid from: 2017-10-17) # min <-1, 0> # max <100, 90000> # info:warning: ncmrwf_tigge_20170807_000.grib2, field 1749 (Snow depth water equivalent, paramId=228141), matched definition (default) => maximum value 78601.367188 is not in [100.000000,15000.000000] |
Installating grib_check.py
Show If |
---|
special | @authenticated |
---|
group | ecmwf |
---|
|
Info |
---|
Show If |
---|
special | @authenticated |
---|
group | ecmwf |
---|
| ECMWF users should get the latest version from the GIT repository. |
|
|
The prerequisite is to have eccodes python 3 modules available in the system. The tool itself consist of two python scripts attached to this page:
The set up file contains mostly a python dictionary with the allowed ranges for parameters. These ranges can be easily tuned and can be specific for specific selection of metadata if needed (e.g. MARS class, level type etc).
Please be aware that this tool is not part of any package yet as is still under development. For example some code refactoring or performance optimisation might be needed.
Image Added