...
The tigge_check can do also very some basic quality control by checking the allowed value ranges for each parameter (with -v option) . Some allowed ranges can become obsolete at some point e.g. due to model upgrade to higher resolution meaning in general different values of some parameters. if they were defined. There is another newer better maintainable tool doing similar basic quality check called grib_check.py (. Read more information about both tools in Data quality checking tools (python source code is available there).
The grib_check.py can be also used for encoding check of another project LC-WFV (Lead Centre for Wave Forecast Verification). The encoding checking is not as comprehensive as in tigge_check e.g. geometry checks are missing completely. Also this tool is not part of ecCodes yet as it is still under development.
Examples of tigge_check usage
...
Checking S2S reforecast data
|
Performance tip to speed up checking big files
There is a new tool (ecCodes v>=2.6.0) called codes_split_file which is useful for parallellising decoding/checking tasks like tigge_check.
...
NAME codes_split_file
DESCRIPTION
Split an input file (GRIB, BUFR etc) into chunks of roughly the same size.
The output files are named input_1, input_2 etc. This is much faster than grib_copy/bufr_copy.
USAGE
codes_split_file [-v] nchunks input
OPTIONS
-v Print the count of messages and files created
If one has a very large inputfile with 1000s of messages, instead of running one process which sequentially checks each message in thefile, one cansplitthefileinto 8 chunks and run the checking code in parallel on the 8 outputfiles.
...
Examples of grib_check.py usage
...
| Code Block |
|---|
python /home/ma/emos/def/lcwfv/bin/grib_check.py -l 131
131 [U component of wind] [min <-250, 5> max <1, 250>] (default)
[min <-150, -10> max <10, 150>] (levtype:pv)
[min <-250, 5> max <-250, 250>] (class:s2, origin:rksl) |
Performance tip to speed up checking big files
There is a new tool (ecCodes v>=2.6.0) called codes_split_file which is useful for parallellising decoding/checking tasks like tigge_check.
NAME codes_split_file The output files are named input_1, input_2 etc. This is much faster than grib_copy/bufr_copy. |
If one has a very large inputfile with 1000s of messages, instead of running one process which sequentially checks each message in thefile, one cansplitthefileinto 8 chunks and run the checking code in parallel on the 8 outputfiles.
| Code Block |
|---|
set -e
# Assume you have 8 cores
codes_split_file 8 my_big.grib
# Now you will have my_big.grib_01, my_big.grib_02, ... my_big.grib_08
for f in my_big.grib_*; do
# Run check in the background. Now multiple processes are running in parallel
tigge_check $f &
done
# With the 'wait' command you can force the execution of the script to pause until a
# all background jobs have finished executing before continuing the execution
# of your script
wait
# Now clean up the split files
rm -f my_big.grib_* |