The CDS and ADS have a regression testing system that can be used to verify that...

  1. The system as a whole is up and working
  2. Individual datasets/adaptors are up and working
  3. Requests deliver the expected files in the expected formats with the expected content. In particular, it can...
    1. Verify that a request returns the expected data (as described by a Python data structure) or
    2. Verify that two stacks return the same data, or
    3. Verify that two different datasets return the same data

Data comparisons are intelligent and, when data is not as expected, the system can specify exactly how it differs from expectation.

The system can be run either from the command line or as part of an ecFlow suite – useful for ad hoc and regular testing respectively. It is designed to bypass the caching system so data retrievals are properly tested.

All the code is in the cds-regression-testing repository.

QUICK START

  1. Clone https://git.ecmwf.int/projects/CDSC/repos/cds-regression-testing/browse.
  2. Copy the cdsapirc files for the stacks you want to use into ~/.cdsapirc_dir
    • Old-system files should be called cdsapirc_<forms_branch>, e.g. cdsapirc_c3sprod
    • New-system (CADS) files be called cdsapirc_<portal>_<JSON_branch>, e.g. cdsapirc_c3s_prod
      • They may have an additional optional "_anything" on the end of them. This is useful if there is more than one stack using the same portal and JSON branch
    • When running the regression_test command (below), which takes stack names as input(s), a stack named X on the command line will use login credentials from ~/.cdsapirc_dir/cdsapirc_X
  3. Then you can run something like
      • "bin/regression_test c3sprod-c3s_prod" to compare old and new stacks. Result files will be compared and the test will fail if they're different.
      • "bin/regression_test c3s_prod" to run tests for just one stack. Result files will be be checked against information on the expected result provided in the test function.
    • These commands will run all explicitly configured tests for datasets that exist in the relevant portal. These are in the tests/test_<dataset_name>.py files.
    • Alternatively you can use the sample.json files as the tests (-k samples), generate random requests with a max size of N as tests (-k random:N) or take real-world user requests from the brokerdb (-k broker:c3sprod)
    • To run just the tests for a subset of datasets use the -d  option. The argument is interpreted as a regex so only a name fragment is required, e.g. "bin/regression_test -d era5 c3s_prod"
    • Use -m to limit the tests to a particular adaptor (e.g. -m mars or -m url )
    • Use -a <dataset>:<testname>  (or -A <dataset>:<testname>) to restart from (or after) a given test. Useful after investigating a failed test.
    • Use -h  to see all options

Content