Bit identical testing

Bit identical

testing over multiple tests can be a very useful tool to ensure that any code changes only impact the results that they are expected to impact, while other results are unaffected. By default,

openifs-test

as described in The Starting the build process section on the OpenIFS 48r1 Getting Started page, only checks that an installation and/or code change builds and runs without any major runtime errors; it does not perform bit-comparison testing against a reference set of known-good output (KGO).

In

order to perform a bit

comparison testing it is first necessary to create the KGO from

a "known-good" source

of the installation of the code. The creation of the KGO should be done prior to any code changes. KGO can be created by running the test with the environment variable IFS_TEST_BITIDENTICAL=init:

IFS_TEST_BITIDENTICAL=init $OIFS_TEST/openifs-test.sh -t 

Note that only -t  is used, so the above assumes that OpenIFS has already been built.

When each test that supports bit identical testing runs, the above will filter out all the norms from the NODE file and write the data to a SAVED_NORMS file. These SAVED_NORMS represent the KGO to which other tests can be compared

Once you have made your changes to source, you can recompile and check that the new norms are identical to the previous ones in SAVED_NORMS

, by setting IFS_TEST_BITIDENTICAL=check:

IFS_TEST_BITIDENTICAL=check $OIFS_TEST/openifs-test.sh -t  

This time, each supported test will compare the norms in the NODE file with the previously created reference in SAVED_NORMS. If they differ then the test will fail. The tests will also fail if there is no reference file because the tests haven't been run in "init" mode.

It can be useful to do this incrementally as you make a series of changes for a single feature that you expect to be bit-identical, to catch any point at which bit-reproducibility is accidentally lost.

It is important to note that running with --clean will wipe any existing SAVED_NORMS; also note that --build-type=BIT and --build-type=DEBUG will produce different norms.

Finally, at present there's currently no built-in mechanism for automatically testing that two different tests produce identical results (e.g. with different namelist settings, or different numbers of MPI tasks or OpenMP threads), although this could be a useful extension in future.