Expand

title	Click here to expand the list of related documents (D1-D5)

Reference ID	Document
D1	CM SAF ATBD: Algorithm Theoretical Baseline Document - HOAPS version 4.0 DOI: 10.5676/EUM_SAF_CM/HOAPS/V002 ATBD_CMSAF_HOAPS_4.0
D2	ValRep: Validation Report SSM/I and SSMIS products - HOAPS version 4.0 DOI: 10.5676/EUM_SAF_CM/HOAPS/V002 ValRep_SSM/I_SSMIS_HOAPS_4.0
D3	Rahpoe, N. (2020) C3S Water Vapour ServiceATBD: Algorithm Theoretical Basis Document. Copernicus Climate Change Service, Document ref. Baseline Document - Total Column Water Vapour brokered from EUMETSAT CM SAF(1988-2014) & ICDR – C3S (2015-2020) from SSM/I & SSMIS measurements C3S_D312b_Lot1.1.3.3-v1.1_202102_ATBD_TCWV_SSMIS_TCDR+ICDR_v1.1.1 TCWV HOAPS: Algorithm Theoretical Basis Document (ATBD) Last accessed on 01.09.2023
D4	Rahpoe, N. (2021) C3S Water Vapour Service: Product Quality Assurance Document. Copernicus Climate Change Service, Document ref. C3S_D312b_Lot1.2.3.	D4	Product Quality Assurance Document - Total Column Water Vapour brokered from EUMETSAT CM SAF(1988-2014) & ICDR – C3S (2015-2020) from SSM/I & SSMIS measurements C3S_D312b_Lot1.2.3.8-v1.1_202102_PQAD_TCWV_SSMIS_TCDR+ICDR_v1.1	D5	Report on Updated KPIs. Key Performance Indicators (KPIs)

Acronyms

1

TCWV HOAPS: Product Quality Assurance Document (PQAD)

Last accessed on 01.09.2023

D5

Meirink, J.F., et al, (2023) C3S cross ECV document

Service: Key Performance Indicators (KPIs). Copernicus Climate Change Service,

Document ref. C3S2_D312a_Lot1.3.7.1_202303_Unified_KPI_Approach_v1.0

Key Performance Indicators (KPIs)

Last accessed: 23.08.2023

Acronyms

Expand

title	Click here to expand the list of acronyms

Acronym	Definition
AIRS	Atmospheric Infra-Red Sounder
CDR	Climate Data Record
CM SAF	Climate Monitoring Satellite Application Facility
COSMIC	Constellation Observing System for Meteorology, Ionosphere, and Climate
DMSP	Defense Meteorological Satellite Program

Expand

title	Click here to expand the list of acronyms

Acronym	Definition
AIRS	Atmospheric Infra-Red Sounder
CDR	Climate Data Record
CM SAF	Climate Monitoring Satellite Application Facility
COSMIC	Constellation Observing System for Meteorology, Ionosphere, and Climate
DMSP	Defense Meteorological Satellite Program
ECMWF	European Centre for Middle-range Weather Forecast
ERA	ECMWF Re-Analysis
HOAPS	Hamburg Ocean & Atmosphere fluxes and Parameters from Satellite data
ICDR	Interim Climate Data Record
IQR	Inter Quartile Range
NASA	National Aeronautics and Space Administration
RMSD	Root Mean Square of Differences
SSM/I	Special Sensor Microwave Imager
SSMIS	Special Sensor Microwave Imager Sounder
TCDR	Thematic Climate Data Record
TCWV	Total Column Water Vapour
TMI	TRMM Microwave Imager
TRMM	Tropical Rainfall Measurement Mission
Water Vapour
TMI	TRMM Microwave Imager
TRMM	Tropical Rainfall Measurement Mission

List of tables

Expand

title	Click here to expand the list of tables

Table 3-1: KPIs for the Water Vapour TCWV TCDR as defined by CMSAF

Table 3-2: The validation results of HOAPS 4.0 TCDR for RSS_SSMI, ERA-Interim, and TMI dataset with their corresponding target requirements fulfillment as defined in Table 3-1

Table 3-3: The validation results of HOAPS 4.0 TCDR (CMSAF) & ICDR (C3S) when compared with RSS_SSMI, ERA-5, and AIRS datasets with their corresponding target requirements fulfillment as defined in Table 3-1

Table 3-4: Results of the hypothesis test upon the number of ICDRs falling outside the 95% confidence interval of the TCDR

List of figures

Expand

title	Click here to expand the list of figures

Figure 3-1: Schematic plot of bias performance of HOAPS 4.0 TCDR (CMSAF) & ICDR (C3S) minus reference sensors with corresponding target requirements

Figure 3-2: Schematic plot of the RMSD performance of HOAPS 4.0 TCDR (CMSAF) & ICDR (C3S) minus reference sensors with corresponding target requirements

Figure 3-3: Schematic plot of stability performance of HOAPS 4.0 TCDR (CMSAF) & ICDR (C3S) minus reference sensors with corresponding target requirements

Scope of the document

This document is the Product Quality Assessment Report (PQAR) for Total Column Water Vapour (product C3S_D312b_Lot1.3.3.14 v1.0 and v1.1) based on SSM/I & SSMIS measurements. It provides a brief guide to the data quality, and describes the validation method.

...

The general picture of validation presented in this document, shows an overall good performance of HOAPS 4.0 TCDR + the C3S ICDR (1988-2020). The target requirements (Table 3-1) set for the inter-comparison of bias and RMSD have been met at the optimal level. The stability has met the optimal level for RSS_SSMI and ERA-5 (see Table 3-3 for further details). The comparison toward AIRS V7 shows a significant drift. The number of ICDR data values outside the 95% interval of TCDRs, is within the expected critical range at 5% significance level, that has been performed with a statistical test for RSS_SSMI and ERA-5. For AIRS, the test rejects the null hypothesis for similarity between TCDRs and ICDRs (Table 43-5).

1. Validated products

The validation includes the CM SAF product TCWV SSM/I and SSMIS from HOAPS 4.0 retrieval, containing gridded monthly mean and 6-hourly total column water vapour data for the period 1988-2020 containing the TCDR (1988-2014) and the follow-up ICDR generated within the C3S project (2015-2020).
For the validation activity, only the monthly mean data sets have been used due to the lack of daily composite data sets as reference, and their consistent statistical representativeness for inter-comparison.

Anchor
validation
validation
2. Reference datasets for validation

The validation of TCDR was primarily based on comparisons with ERA-Interim reanalysis, COSMIC (beta-version, ROM SAF), RSS_SSMI (SSM/I+SSMIS) V7, and TMI V7. The reference datasets are described and discussed in Sections 4.5-4.7 of CM SAF Validation Report [D2].

...

RSS_SSMI V7 (1988/01-2020/12)
ERA-5 (1988-2020/12)
AIRS V7 (2002/09-2020/12)

Anchor
validation
validation
3. Validation results

3.1 TCDR performance

The validation methodology is outlined in Section 6.1 [D2]. The results for total column water vapour are presented in Section 6.7.1 and further discussed in Section 6.7.2 of the CM SAF Validation Report [D2].

For TCWV, the target requirements are listed in Table 3-1.

The HOAPS 4.0 validation report [D2, Section 6.7.2] gives the following summary of quality. The HOAPS-4.0 monthly mean TCWV data show the following absolute bias and RMSD results, when compared against ERA-Interim and the satellite-based RSS_SSM and TMI products (Table 3-2):

average (absolute) biases of <0.4 kg/m² and
RMSD of ≤1.1 kg/m²

...

The decadal stability of HOAPS 4.0 is 0.00±0.008 kg/m²/dec, which fulfills the requirements for ‘optimal category’ (<0.08 kg/m²/decacde) as described in Section 7 [D2].

Anchor

	table3_1
	table3_1	table1	table1

Table 3-1: KPIs for the Water Vapour TCWV TCDR as defined by CMSAF (see Table 6-6 in [D2])

Category	Bias [kg/m²]	RMSD [kg/m²]	Stability [kg/m²/decade]
Threshold	3	5	0.4
Target	1.4	2	0.2
Optimal	1.0	1	0.08

The SSM/I and SSMIS 6-hourly daily composites fulfil the GCOS frequency requirement of 4-hourly observations when input data from different DMSP satellites are considered.

On the other hand, the spatial resolution of 50 km x 50 km does not fulfill the spatial resolution requirement set by GCOS (25 km), which is the only limitation of the data set.

Anchor

	table3_2
	table3_2	table2	table2

Table 3-2: The validation results of HOAPS 4.0 TCDR for RSS_SSMI, ERA-Interim, and TMI dataset with their corresponding target requirements fulfillment as defined in Table 3-1. The numbers are taken from the CM SAF Validation Report [D2] Section 6.7.1 - 6.7.2 respectively.

Reference Dataset	Bias [kg/m²]	RMSD [kg/m²]	Stability [kg/m²/decade]
RSS_SSMI: 1988-2014	-0.35 optimal	0.8 optimal	0.00±0.008 optimal
ERA-Interim: 1988-2014	0.3 optimal	1.1 target	-
TMI: 1998-2014	-0.39 optimal	1.0 optimal	-

3.2 TCDR+ICDR performance

The key performance for TCDR+ICDR is calculated in the same manner as for TCDR. The validation metrics, i.e. bias, RMSD, stability, KPI test on ICDRS vs. TCDR with corresponding error estimates on bias and stability have been calculated for RSS-SSMI, ERA-5, and. The details on methods to derive the key validation metrics are described in Section 3 of PQAD [D4]. [D4] also contains detailed results of the TCDR+ICDR quality assessment and only a summary is provided here.

The summary of inter-comparison of HOAPS 4.0 TCDR+ICDR (1988-2020) is shown in Table 3-3 and Figures 3-1 - 3-3.

The overall bias (< 1 kg/m²) and RMSD (< 1 kg/m²) fulfil the optimal requirement for all three reference sensors. The largest bias is observed between HOAPS and ERA-5 (0.56 kg/m²). On the other hand the stability is fulfilled at the optimal level for RSS_SSMI at 100% and ERA-5 at 74%, respectively. The comparison with AIRS V7 shows significant increase of the trend of the bias (drift), where even the threshold requirement is violated. The reason for the drift is the synthetic micro-wave data that have been included in the retrieval scheme.

Anchor

	table3_3
	table3_3

Table 3-3: The validation results of HOAPS 4.0 TCDR (CMSAF) & ICDR (C3S) when compared with RSS_SSMI, ERA-5, and AIRS datasets with their corresponding target requirements fulfillment as defined in Table 3-1. In brackets the numerical values of probability coverage of requirements are shown if these are lower than 100%.

Reference Dataset	Bias [kg/m²]	s_bias[kg/m²]	RMSD [kg/m²]	Lag-1 Autocorrelation r(1)	Stability (via s) [kg/m²/decade]	Stability (via spread) [kg/m²/decade]
RSS_SSMI: 1988/01-2020/12	-0.28 Optimal	0.14	0.31 Optimal	0.85	0.033±0.007 Optimal	0.033±0.010 Optimal
ERA-5: 1988/01-2020/12	0.56 Optimal	0.63	0.85 Optimal	0.63	0.067±0.020 Optimal (74%) Target	0.067±0.028 Optimal (68%) Target
AIRS: 2002/09-2019/12	0.20 Optimal	0.41	0.45 Optimal	0.81	0.651±0.029 Not met	0.651±0.031 Not met

Image Added

Anchor

figure1figure1

Image Removed

	figure3_1
	figure3_1

Figure 3-Figure 1: Schematic plot of bias performance of HOAPS 4.0 TCDR (CMSAF) & ICDR (C3S) minus reference sensors with corresponding target requirements.

Image Added

Anchor

figure2figure2

Image Removed

	figure3_2
	figure3_2

Figure 3-Figure 2: Schematic plot of the RMSD performance of HOAPS 4.0 TCDR (CMSAF) & ICDR (C3S) minus reference sensors with corresponding target requirements.

Image Added

Anchor

	figure3_3
	figure3

Image Removed
_3
Figure 3-3: Schematic plot of stability performance of HOAPS 4.0 TCDR (CMSAF) & ICDR (C3S) minus reference sensors with corresponding target requirements.

3.1 Results

The KPI test of ICDRs vs. TCDRs leads to similar conclusions. The ICDR differences relative to the RSS_SSMI and ERA-5 validation datasets stay in the expected range for TCDRs (Table 3-4). Only 5 out of 72 ICDRs show larger differences, which is expected from the 95% confidence interval. On the other hand, due to large drift between HOAPS 4.0 SSMI(S) and AIRS V7, the number of ICDR failures (K=30) exceeds the critical number of 7. Hence the ICDR differences between HOAPS & AIRS V7 do not fulfill the null hypothesis of similarity between ICDRs and TCDR. Further details can be found in the PQAD [D4] and Report on Updated KPI's [D5].

Anchor

	table3_5
	table3_5	table4	table4

Table 3-4: Results of the hypothesis test upon the number of ICDRs falling outside the 95% confidence interval of the TCDR.

Reference Dataset	TCDR lower threshold 2.5% [kg/m²]	TCDR upper threshold 97.5% [kg/m²]	Number of ICDRs	Critical number of failures at 5% rate	Observed K failures of ICDRs outside the 95% interval [2.5%,97.5%]	Cumulative probability P(N-K,N,95%) P >5% : Accept P < 5%: Reject
RSS_SSMI: 1988/01-2020/12	-0.52	0.00085	72	K > 7	5	29% Accept
ERA-5: 1988/01-2020/12	-0.6	1.7	72	K > 7	5	29% Accept
AIRS: 2002/09-2019/12	-0.28	0.62	72	K > 7	30	0% Reject

The general picture of validation summarized here and in the PQAD [D4], shows an overall good performance of HOAPS 4.0 TCDR (CMSAF) + ICDR (C3S) (1988-2020) toward the reference sensors/datasets RSS_SSMI and ERA-5.

Info

icon	false

_{This document has been produced with funding by the European Union in the context of the Copernicus Climate Change Service (C3S)}

.The activities leading to these results have been contracted

_{, operated by the European Centre for Medium-Range Weather Forecasts}

, operator of C3S

_{on behalf}

of

_{on the European Union (}

Delegation agreement

_{Contribution Agreement signed on}

11

_22/

11

_07/

2014

_2021).

_{All information in this document is provided "as is" and no guarantee}

or

_{of warranty is given that the information is fit for any particular purpose.}

_{The users thereof use the information at their sole risk and liability. For the avoidance of all doubt, the European Commission and the European Centre for Medium-Range Weather Forecasts have no liability in respect of this document, which is merely representing the author's view.}

Space shortcuts

Page tree

Versions Compared

Old Version 5

New Version 6

Key

Acronyms

Acronyms

List of tables

List of figures

Scope of the document

1. Validated products

Anchor
validation
validation
2. Reference datasets for validation

Anchor
validation
validation
3. Validation results

3.1 TCDR performance

3.2 TCDR+ICDR performance

3.1 Results

Related articles

Space shortcuts

Page tree

Page History

Versions Compared

Old Version 5

New Version 6

Key

Acronyms

Acronyms

List of tables

List of figures

Scope of the document

1. Validated products

Anchorvalidationvalidation2. Reference datasets for validation

Anchorvalidationvalidation3. Validation results

3.1 TCDR performance

3.2 TCDR+ICDR performance

3.1 Results

Related articles

Anchor
validation
validation
2. Reference datasets for validation

Anchor
validation
validation
3. Validation results