Contributors: Hans Gleisner (DMI)
History of modifications
List of datasets covered by this document
Related documents
Acronyms
List of tables
List of figures
General definitions
Scope of the document
This document is the Product Quality Assessment Report (PQAR) for the Gridded Tropospheric Humidity Profiles (product C3S_D312b_Lot1.3.3.9) based on GNSS Radio Occultation (RO) measurements. It provides references to the quality assessment (validation and operational quality monitoring) done as a part of the normal ROM SAF operations, and documented in the ROM SAF Validation Reports [D1-3]. The product quality assessment is based on methods referenced in the Product Quality Assurance Document.
Executive summary
The project C3S_312b_Lot1 includes brokering of gridded monthly-mean tropospheric humidity data from the EUMETSAT ROM SAF. This document refers to the ROM SAF documentation [D1,2] describing the product validation methodology and validation results, including the main validation results in graphical form.
The ROM SAF documentation covers both the CDR (ROM SAF product GRM-29-L3-H-R1) and the ICDR (ROM SAF product GRM-29-L3-H-I1). The CDR covers the time period December 2006 to December 2016, while the ICDR covers the time period from January 2017 and onward, being regularly updated with a three-month delay. The regularly updated ICDR has the same characteristics as the CDR, which has been generated in a ROM SAF reprocessing activity.
1. Product validation methodology
The validation includes the ROM SAF products GRM-29-L3-H-R1 and GRM-29-L3-H-I1, both containing gridded monthly mean tropospheric humidity data and associated variables. These products are described by the ROM SAF Product User Manual [D4].
The validation methods are described in the beginning of Section 4 of the ROM SAF Level 3 Validation Report [D2], and the actual requirements are stated in Section 6 and in Annex A of the same document.
2. Validation results
The main validation results are given in Section 5.2 (Figure 22a) of the ROM SAF Level 3 Validation Report [D2]. Additional validation results, relevant for the quality assessment are shown in Section 4.2.2 (Figure 8e) and in Section 5.1 (Figure 20e) of the same report [D2].
The ROM SAF ICDR time series are mainly validated regarding the stability in time. In the ROM SAF ICDR validation report [D3], the continuity across the CDR-ICDR and the ICDR v1.0-v1.1 transitions are evaluated. The results relevant for the humidity data are Figures 3 and 6 in that report [D3].
3. Application(s) specific assessments
In addition to the extensive product validation (see chapter 2 for results and chapter 2/3 in [D6] for validation methodology) a second assessment is introduced to evaluate the Interim Climate Data Record (ICDR) against the Thematic Climate Data Record (TCDR) in terms of consistency. Since frequent ICDR deliveries make detailed validation not feasible, a consistency check against the deeply validated TCDR is used as an indication of quality. This is done by a comparison of the following two evaluations:
- TCDR against a stable, long-term and independent reference dataset
- ICDR against the same stable, long-term and independent reference dataset
The evaluation method is generated to detect differences in the ICDR performance in a quantitative, binary way with so called Key Performance Indicators. The general method is outlined in [D5] chapter 3. The same difference between TCDR/ICDR and the reference dataset would lead to the conclusion that TCDR and ICDR have the same quality (key performance is "good"). Variations or trends in the differences (TCDR/ICDR against reference) would require a further investigation to analyze the reasons. The key performance would be marked as "bad". The binary decision whether the key performance is good or bad is made in a statistical way by a hypotheses test (binomial test). Based on the TCDR/reference comparison (global means, monthly or daily means) a range is defined with 95% of the differences are within. This range (2.5 and 97.5 percentile) is used for the ICDR/reference comparison to check whether the values are in or out of the range. The results could be the following:
- All or a sufficient high number of ICDR/reference differences lies within the range defined by the TCDR/reference comparison: Key performance of the ICDR is "good"
- A smaller number of ICDR/reference differences is within the pre-defined range: Key performance of the ICDR is "bad"
3.1 Results
The results of the KPI test are summarized in Table 3-1.
Table 3-1: Summary of KPI results with 2.5 and 97.5 percentiles and number of ICDR months within the range. Colors green or red mark the results of the binomial tests as good or bad, respectively.
| Percentiles | Tropospheric Humidity Profiles | ||||
|---|---|---|---|---|---|
p2.5 | p97.5 | 0-4 km | 4-8 km | 8-12 km | |
| 01/2017 - 02/2021 | 0-4 km = -2% 4-8 km = -1.04% 8-12 km = -0.66% | 0-4 km = -0.93% 4-8 km = 0.82% 8-12 km = 2.13% | 14/51 | 47/51 | 46/51 |
| 01/2017 - 08/2021 | 17/57 | 51/57 | 53/57 | ||
| 01/2017 - 02/2022 | 21/63 | 55/63 | 57/63 | ||
| 01/2017 - 08/2022 | 28/69 | 62/69 | 54/69 | ||
| 01/2017 - 02/2023 | 30/75 | 68/75 | 70/75 | ||
Percentiles were calculated based on the comparison of the TCDR against ERA-Interim as reference dataset for the tropospheric humidity in three different height levels: 0-4 km, 4-8 km and 8-12 km.
Especially the lowest atmospheric layer ICDR months are mostly outside the pre-defined percentiles and result in "bad" KPI tests. This is mainly related to a known bias shift in the reference data (ERA-Interim). There was a change in the ERA-Interim data assimilation in 2017, which was close to the TCDR/ICDR transition. In addition, the reference data was changed in August 2019 from ERA-Interim to ERA5 leading to differences without having a significant impact. The ICDR becomes closer to compliance with a de-trend before the statistical tests, again without beeing on a significant level. The next reprocessing of all available Metop data is planned to be released by ROM SAF in 2024 and expected to solve the inhomogeneities between TCDR and ICDR.
4. Compliance with user requirements
The main validation results are discussed in Sections 4.2.5, 5.3, and 7 of the ROM SAF Level 3 Validation Report [D2], followed by a formal statement of compliance.
References
References are found in Section 1.2 of the ROM SAF Level 3 Validation Report [D2] and in Section 1.2 in the ROM SAF ICDR Validation Report [D3].
This document has been produced with funding by the European Union in the context of the Copernicus Climate Change Service (C3S), operated by the European Centre for Medium-Range Weather Forecasts on behalf on the European Union (Contribution Agreement signed on 22/07/2021).
All information in this document is provided "as is" and no guarantee of warranty is given that the information is fit for any particular purpose.
The users thereof use the information at their sole risk and liability. For the avoidance of all doubt, the European Commission and the European Centre for Medium-Range Weather Forecasts have no liability in respect of this document, which is merely representing the author's view.
