grib_compare: index based comparison
Step-by-step guide
Suppose you have two large GRIB files and their messages are not in order, how do you compare them? One technique is to use the "-r" key for grib_compare. This will compute the md5 hash value of each message (meta-data only) and sort both files (in memory) by that md5 value before doing the comparison.
Another technique is to build index files from both inputs (using grib_index_build) and then compare those index files. This can be considerably faster. The user has to decide on which keys to build the index files. An example is shown here.
Here is the version with "-r":
% grib_compare -r $input1 $input2
And now using grib_index_build with MARS keys:
#!/bin/sh grib_index_build -o idx1 $input1 >/dev/null grib_index_build -o idx2 $input2 >/dev/null grib_compare idx1 idx2 rm -f idx1 idx2 # Clean up
By default grib_index_build uses the MARS keys (those in the "mars" namespace). Otherwise the user can pass his/her own desired keys via the "-k" option.
An even faster version of the 2nd script can be done if we launch the two grib_index_build processes in the background (run them concurrently):
#!/bin/sh grib_index_build -o idx1 $input1 >/dev/null & grib_index_build -o idx2 $input2 >/dev/null & wait # For the background tasks for finish grib_compare idx1 idx2 rm -f idx1 idx # Clean up