Versions Compared
Key
- This line was added.
- This line was removed.
- Formatting was changed.
Bit identical testing
Bit identical testing over multiple tests can be a very useful tool to ensure that any code changes only impact the results that they are expected to impact, while other results are unaffected. By default, openifs-test as described
...
in The Starting the build process section on the OpenIFS 48r1 Getting Started page, only checks that an installation and/or code change builds and runs without any major runtime errors; it does not perform bit-comparison testing against a reference set of known-good output (KGO).
In order to perform a bit comparison testing it is first necessary to create the KGO from a "known-good" source of the installation of the code. The creation of the KGO should be done prior to any code changes. KGO can be created by running the test with the environment variable IFS_TEST_BITIDENTICAL=init:
Code Block | ||||
---|---|---|---|---|
| ||||
IFS_TEST_BITIDENTICAL=init $OIFS_TEST/openifs-test.sh -t |
Note that only -t
is used, so the above assumes that OpenIFS has already been built.
When each test that supports bit identical testing runs, the above will filter out all the norms from the NODE
file and write the data to a SAVED_NORMS
file.
These SAVED_NORMS
represent the KGO to which other tests can be compared
Once you have made your changes to source, you can recompile and check that the new norms are identical to the previous ones in SAVED_NORMS
, by setting IFS_TEST_BITIDENTICAL=check:
Code Block | ||||
---|---|---|---|---|
| ||||
IFS_TEST_BITIDENTICAL=check $OIFS_TEST/openifs-test.sh -t |
This time, each supported test will compare the norms in the NODE file with the previously created reference in SAVED_NORMS
. If they differ then the test will fail. The tests will also fail if there is no reference file because the tests haven't been run in "init
" mode.
It can be useful to do this incrementally as you make a series of changes for a single feature that you expect to be bit-identical, to catch any point at which bit-reproducibility is accidentally lost.
It is important to note that running with --clean
will wipe any existing SAVED_NORMS
; also note that --build-type=BIT
and --build-type=DEBUG
will produce different norms.
Finally, at present there's currently no built-in mechanism for automatically testing that two different tests produce identical results (e.g. with different namelist settings, or different numbers of MPI tasks or OpenMP threads), although this could be a useful extension in future.
Panel | ||
---|---|---|
| ||
On this page ...
|
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||
|