Last modified on Feb 28, 2024 16:11

Contributors: J. El Kassar (FUB), R. Preusker (FUB)

Issued by: Rene Preusker / Jan R. El Kassar

Date: 31/08/2020

Ref: C3S_312b_Lot1.2.3.5-v1.1_202008 _PQAD_v1.1.2

Official reference number service contract: 2018/C3S_312a_Lot1_FUB/SC1

History of modifications

List of datasets covered by this document

Related documents

Acronyms

General definitions

In the scope of the Copernicus Climate Change Service (C3S), a Thematic Climate Data Record (TCDR) has a fixed end point, whereas an Interim Climate Data Record (ICDR) is extended continuously. The TCWV_MERIS_SSMI_TCDR L3 product is a TCDR which consists of two satellite-based estimates of the Total Columnar Water Vapor (TCWV) of the atmosphere for the time period 2002-2012. The two estimates are derived by two different instruments over two types of surface. Over land, data from the Medium Resolution Imaging Spectrometer (MERIS) are used. Over water, the Special Sensor Microwave/Imager (SSM/I) is used.

All documents related to the TCWV_MERIS_SSMI_TCDR product are mostly limited to the processing of L2 MERIS data by the Freie Universität Berlin (FUB). SSM/I L2 data have been provided by the German Weather Service (DWD) and were not part of the processing at FUB and are thus not discussed extensively in this or any of the other supplementary documents. The validation results summary for SSMI/S L2 is included in this document, but extensive description of the validation method & procedure for L2 SSMI/S dataset can be found in [D1] already.

Scope of the document

This Product Quality Assurance Document (PQAD) describes the product validation methodology of the Level 2 and Level 3 Total Column Water Vapor (TCWV) datasets from the Medium Resolution Imaging Spectrometer (MERIS) produced within C3S_312b Lot1 by the Freie Universität Berlin (FUB) and the SSMI/S dataset produced within the HOAPS by the Deutsche Wetterdienst (DWD). This Climate Data Record consists of only one merged product: the integrated water vapor content of the atmosphere.

Executive summary

The TCDR of Total Column Water Vapour (TCWV) L2 from the Freie Universität Berlin (FUB) is an in-house product delivered to the Climate Data Store (CDS) of the Copernicus Climate Change Service (C3S).

The MERIS TCWV retrieval is based on the Cloud Aerosol and Water Vapor algorithm (CAWA) which was developed at the FUB [Preusker et al., 2015] within the Advanced Clouds, Aerosols and Water Vapour products for Sentinel 3/OLCI (CAWA) ESA Scientific Exploitation of Operational Missions Element Program (SEOM). The retrieval is applied on the MERIS instrument onboard the polar orbiting ENVISAT.

The SSMI TCWV L2 is provided by DWD and is retrieved by SSM/I and SSMIS instruments onboard several polar orbiting DMSP satellites. For further information on the SSM/I SSMIS dataset we refer to Graw et al. (2017).

The MERIS TCWV and SSMI TCWV datasets are combined by the FUB team and comprises of 10 years (2002-2012) L3 monthly mean time series. The CDR provided here includes monthly means of TCWV on a regular global latitude-longitude grid, merged from MERIS (over land) and SSM/I (over ocean).

The intercomparison of MERIS TCWV (over land) with the in-situ measurements leads to a mean absolute difference (MAD) < 1.9 kg/m² and bias corrected root mean square deviation (RMSD) of < 2.5 kg/m² (See Table 1).

Within the CMSAF an intercomparison of SSMI/S TCWV (over ocean) with ERA-Interim has been proceeded for the time period 1982-2014 leading to a global mean bias of 0.3 kg/m² [D1 – Chap. 6.7]. As it is described in the validation report:

‘The largest differences occur over subtropical dry regions, where ERA-Interim is considerably lower compared to HOAPS-4.0. Here, the positive bias exceeds 2.5kg/m². Smaller but still positive differences are present around Antarctica. Largest negative differences occur over the storm track regions. The global mean RMSD is 1.1kg/m².’

1. Validated products

The MERIS_SSMI_TCWV contains one variable: the integrated atmospheric water vapor content. This variable consists of products from two different sensors which should be looked at separately and combined. Furthermore the MERIS TCWV product can be evaluated at different stages of the processing. Comparison with ground-truth (i.e. stations that measure TCWV in situ) should be done at the Level 2 stage, where TCWV is still stored in the so-called satellite projection. Validation will also be performed with the merged Level 3 product.

1.1 MERIS TCWV (Level 2, swath-projection)

The first level of processing with the CAWA algorithm produces MERIS Level 2 TCWV. These are estimated from MERIS Level 1b Reduced Resolution Radiances with a pixel-wise resolution of ca. 1 by 1 km. This resolution is ideal for validation against ground stations, since these mostly only represent the TCWV value for a very limited area.

1.2 MERIS TCWV (Level 3, regular grid)

In order to also validate the final product provided to the user, we also validate the monthly means of MERIS SSMI TCWV against ground stations (Table 3). In this case we will compare monthly means at the high resolution (0.05°) to ground stations around the world.

2. Description of validating datasets

2.1 Atmospheric Radiation Measurement (ARM) Climate Research Facility

The ARM Climate Research Facility [Stokes and Schwartz, 1994] consists of a global system of several fixed and mobile stations which employ a wide array of ground-based instruments to observe the atmospheric processes such as movement of aerosols and cloud formation. In order to quantify the amount of water vapor in the atmosphere all mobile and fixed ARM sites are equipped with Microwave Radiometers (MWR). These instruments measure emitted radiation at two channels 23.8 and 31.4 GHz. The retrieved brightness temperatures are converted to units of Liquid Water Path (LWP) and Precipitable Water Vapor (PWV). LWP is the amount of liquid water in the atmosphere, whereas PWV corresponds with TCWV, the integrated content of water vapor in the atmosphere.

The algorithm used for ARM TCWV is an iterative physical retrieval based on radiosonde profiles as a priori in combination with a statistical approach which uses monthly retrieval coefficients. Additionally, the measured brightness temperatures are corrected with a static offset which is determined yearly. The quality of the ARM water vapor product is considered to be very high and they provide continuous datasets for several permanent stations, as well as shorter datasets for many mobile stations.

2.2 Aerosol Robotic Network (AERONET)

AERONET [Holben et al., 1998] is a network that uses sun-photometry to retrieve atmospheric parameters. Originally it was established to study columnar aerosol properties. It uses the principle of water vapour absorption around the 940 nm absorption band and estimates TCWV from the transmittance. Due to the simplicity of the retrieval and the technology behind the sun-photometer it is widespread and can provide good global data coverage in real time. 

2.3 SUOMINET Global Ground-based Global Navigation Satellite System (GNSS) TCWV

The SUOMINET GNSS network uses the principle of radio-occultation to obtain atmospheric properties [Ware et al., 2000]. It is a university-based, real-time, international network that mainly uses the Global Positioning System (GPS).

The basic principle relies on the fact, that the radio signal sent by GPS satellites is delayed by the atmosphere, mainly water vapor. The station receives the GPS signal and calculates from the delay in the radio signal a refractive index and from the refractive index a concentration of water vapour. Since the GPS satellites are moving, the sounding of the atmosphere is possible as well as the retrieval of integrated water vapor.

The SUOMINET dataset provides real-time data in a continuous time series for many stations around the world.

2.4 Global Climate Observing System (GCOS) Upper-Air Network (GUAN)

The GUAN dataset is based on radiosonde observations of standard meteorological parameters such as temperature and humidity. GUAN provides long time series for many stations spread all over the earth with at least one observation per day for standardized times. It provides long time series that date back to the late 1940s. This spatial coverage and timeliness makes it formidable for daily or monthly validation of the TCWV product.

2.5 GCOS Reference Upper-Air Network (GRUAN)

Similar to the GUAN, GRUAN also employs radiosondes to measure meteorological variables . [Immler et al., 2010]. However, this network goes far beyond measuring standard meteorological parameters. GRUAN employs stricter constraints on, e.g., the type of radiosonde used. Thus it provides more reliable and precise vertical profiles, as well as integrated parameters, compared to GUAN.

Its objective is to provide long-term, high quality upper-air climate records, for the use as reference in global observing systems (e.g. satellite retrievals) [Bodeker et al., 2015]. GRUAN stations measure temperature, humidity, aerosols and ozone. Apart from launching radiosondes, GRUAN also employs ground-based instruments such as LIDAR (light detection and ranging) or GNSS (described in Section 2.3) in parallel.

3. Description of product validation methodology

According to Loew et al. (2017) validation can be defined as '(1) the process of assessing, by independent means, the quality of the data products derived from the system outputs and (2) confirmation, through the provision of objective evidence, that specified requirements, which are adequate for an intended use, have been fulfilled'. Both aspects are addressed here, and the 'specified requirements' are given by the Key Performance Indicators (KPIs). In the validation process the following five steps can be distinguished.
The first step is quality checking. It involves the selection of data to enter the validation process using available quality information. This holds for satellite products, reference data and ancillary data.
The second step is spatial-temporal collocation. Various constraints must be considered. In particular, satellite products and reference data should have comparable space/time sampling, and comparable space/time resolution. Another, sometimes conflicting, requirement is that sufficient statistics should be gathered.

The third step is homogenization. It includes conversions and e.g. application of averaging kernels needed to make the satellite and reference data comparable. After this step, a set of satellite measurements (x = xi, i=1…N) and a corresponding set of reference data (y = yi) are available for quantitative comparison. The (xi,yi) are often instantaneous measurements, but here they refer to gridded and aggregated products: the KPIs are in most cases defined for monthly-mean global-mean quantities. Since aggregated products are considered, the second step in the validation process (spatial-temporal collocation) will not be entirely feasible and resulting uncertainties should be taken into account.

The fourth step is the calculation of metrics quantifying the consistency between satellite products and reference observations.

3.1 Quality Checking

Quality checking for the L2 satellite data is done by filtering data according to flags. Flags are markers for each pixel in the satellite swath which contain information about the surface type (e.g. land, water or cloud), the quality (invalid measurement by the satellite, failed retrieval, etc.) or specific contaminants (e.g. high aerosol loads, sun glint).
In this version of the PQAD for TCWV_MERIS_SSMI_TCDR we validate for MERIS TCWV retrieval only over land surfaces. Thus we filtered out all pixels with water surfaces, clouds and high aerosol loads. To assure that the retrieval is not contaminated by cloud shadows or undetected neighboring clouds we added a margin of 6 pixels to the cloud mask.

For the reference datasets the supplied quality masks were used to filter out invalid or contaminated retrievals.
There was no specific quality control for L3 satellite observations, the associated flags for a grid box were used to sort out missing or bad data. The reference dataset was filtered according to supplied quality masks.

3.2 Spatio-temporal Collocation and Homogenisation

We took two different approaches for the collocation: the first approach is based on individual, instantaneous match-ups of Level 2 (in the irregular swath-projection) for the years 2009 to 2011. The second approach is done with monthly observations from the finished TCWV dataset in the high resolution (0.05°) compared against monthly observation from the GUAN network for the whole time period of the dataset from May 2002 to March 2012.

3.2.1 Individual, instantaneous Level 2 match-ups

The spatio-temporal collocation was done by taking a 7 x 7 km area (7 x 7 Pixels in reduced resolution), equivalent to a ca. 3km radius around a ground station. The overpass of the satellite needs to be in a ±15 min window. For these filters collocation mismatch uncertainty is considered to be negligible. As an additional constraint, in case there were less than 50% of valid MERIS pixels, the measurement was also rejected. Outliers, deviating more than 3 σ were also discarded.
We used this set of filters for ARM, AERONET and SUOMI GNSS.

The collocation criteria for GRUAN were set more loosely, since collocation in between a ±15 min timeframe was rarely achieved. Furthermore, radiosondes can drift very far from their original starting point. Thus, we adapted a radius of ca. 20 km around the GRUAN stations and a time window of ±3 h, which increases collocation uncertainty.

In Figure 1 you can see the spatial distribution. While ARM, GNSS and AERONET are spread out over the globe, GRUAN stations are more limited to certain regions. This, however, had no significant impact on the validation results.

Figure 1: All stations and their associated networks that were used for the match-up validation of the MERIS TCWV dataset between 2009 and 2011.

3.2.2 Level 3 Monthly observations

The comparison of monthly, gridded observations against the GUAN dataset, the spatio-temporal collocation is done by taking 0.05° x 0.05° grid boxes that correspond with the positions of the GUAN dataset. If a grid box collocated with a GUAN station contains at less than 15 daily satellite observations or less than 15 daily radiosonde observations the grid box will not be included in the validation.

Daily radiosonde observations were achieved by averaging over all radiosondes flown per day. These daily observations were then averaged into a monthly mean. In total, 960 GUAN radiosonde stations are included in the validation of Level 3 monthly data. The locations of all stations that were used can be seen in Figure 2.

Only GUAN stations were compared against the whole time period of the Level 3 dataset, since the other dataset were either limited in spatial distribution or did not have the temporal coverage to cover the whole 10 year period of the dataset. AERONET would have been suitable as well, but we chose not to use the results, since AERONET compared to radiosondes, MWR and GPS shows a high dry bias (Ramírez et al., 2014).

Figure 2: All GUAN stations that were used for the match-up validation of the Level 3 SSMI MERIS TCWV dataset between May 2002 and March 2012.

3.3 Validation Metrics

In this section the used validation metrics are presented.

3.3.1 Bias

The common metric for systematic differences is the bias b:

where E is the expectation operator.

3.3.2 Root-mean-square-deviation (RMSD) and centered RMSD

To measure statistical spread, the root-mean-square deviation (RMSD) is commonly used:

Often the RMSD is corrected for the bias between the datasets, yielding the bias-corrected or centered RMSD (cRMSD):

3.3.3 Mean Absolute Deviation (MAD)

In some cases the absolute value of deviations rather than their square, as in the RMSD, is taken. The resulting metric is the mean absolute deviation (MAD):

3.3.4 Pearson Correlation Coefficient and Linear Regression Coefficients

In order to quantify the statistical dependency between the datasets the linear (Pearson) correlation coefficient and the offset and slope of the linear best fit are calculated.

3.3.5 Temporal Stability

Finally, the temporal stability β is defined as the change in the bias of a data set over time:

It can be estimated by linear regression analysis of the time series of . Before estimating the stability, it is important to first deseasonalize the datasets and check for breakpoints, which will be part of the Product Quality Assessment Report (PQAR) [D2]

3.4 Specified Requirements – the KPIs

The KPIs are defined targets of precision and accuracy for a given dataset. For TCWV, reliable target requirements have been set by CMSAF in the Validation Report for the HOAPS release 4.0 [D1]. The requirements are based on several independent studies (GCOS, WMO-OSCAR) and can be found in Table 1.

Table 1: KPIs as defined by CMSAF:

In our case, the Bias is a metric of accuracy, while the cRMSD is a measure of the precision of the dataset. The three different categories (threshold, target, optimal) refer to certain levels of quality. In the best case the "optimal" criteria would be met. Above these levels, no further improvement is necessary. In order to have a useful dataset at least the "threshold" requirements should be met. "Target" requirements are intermediate and would result in significant improvement, if reached.

4. Summary of validation results

Here, you can find some preliminary validation results for the MERIS TCWV dataset. More detailed information on the validation results can be found in the Product Quality Assessment Report (PQAR) [D2]. In Table 2 we show the summary of the instantaneous match-up validation of MERIS Level 2 TCWV. In Table 3 we show the validation results for the comparison between Level 3 SSMI MERIS TCWV and monthly GUAN radiosonde observations.

Table 2: Results of Level 2 match-up intercomparison analysis for different validation metrics.

Table 3: Results of Level 3 monthly obsvertaion intercomparison analysis for different validation metrics.

In comparison with the reference datasets the MERIS_SSMI_TCWV dataset meets all threshold criteria (as defined in Table 1) and in comparison with ARM and GRUAN even the target requirements are met.

Further research can be invested into finding out why cRMSD is higher for the Level 3 dataset.

Nonetheless we could demonstrate that this TCWV dataset is formidable and meets all requirements. A more detailed look on the validation results as well as figures can be found in the PQAR [D2].

4.1 SSMI/S L2 validation results

For SSMI/S L2 dataset the validation results from the validation report of CMSAF HOAPS 4.0 [D1] are summarized in Table 4.

The comparison has been performed for two products, i.e. ERA-Interim and RSS-TMI (Remote Sensing System-TRMM Microwave Imager).  According to validation results for SSMI/S  (See Table 4), the optimal requirements for bias and cRMSD (as defined in Table 1) are met by  SSMI/S L2 total column water vapour product toward comparison products. Further details on validation method can be found in the [D1] document.

Table 4: Results of Level 2 global mean intercomparison analysis for SSMI/S (HOAPS 4.0) taken from [D1].

References

Graw K., A. Andersson, M. Schröder, K. Fennig, 2017: Algorithm Theoretical Baseline Document HOAPS version 4.0, DOI: 10.5676/EUM_SAF_CM/HOAPS/V002

Holben B.N., T.F.Eck, I.Slutsker, D.Tanré, J.P.Buis, A.Setzer, E.Vermote, J.A.Reagan, Y.J.Kaufman, T.Nakajima, F.Lavenu, I.Jankowiak, A.Smirnov, 1998: AERONET—A Federated Instrument Network and Data Archive for Aerosol Characterization

Immler F. J., J. Dykema, T. Gardiner, D. N. Whiteman, P. W. Thorne, and H. V, 2010: Reference Quality Upper-Air Measurements: guidance for developing GRUAN data products

Loew A., W. Bell, L. Brocca, C. E. Bulgin, J. Burdanowitz, X. Calbet, R. V. Donner, D. Ghent, A. Gruber, T. Kaminski, J. Kinzel, C. Klepp, J.C. Lambert, G. Schaepman‐Strub, M. Schröder, T. Verhoelst, 2017: Validation practices for satellite-based Earth observation data across communities, Rev. Geophys., 55, 779–817, doi:10.1002/2017RG000562

Pérez-Ramírez, D., D. N. Whiteman, A. Smirnov, H. Lyamani, B. N. Holben, R. Pinker, M. Andrade, and L. Alados- Arboledas (2014), Evaluation of AERONET precipitable water vapor versus micro- wave radiometry, GPS, and radiosondes at ARM sites, J. Geophys. Res. Atmos., 119, 9596–9613, doi:10.1002/2014JD021730.

Preusker R., H. Diedrich and J. Fischer, 2015: Retrieval of Total Column Water Vapor from MERIS/OLCI and MODIS above Land- and Ocean Surfaces ATBD

Stokes G. M. and S. E. Schwartz, 1994: The Atmospheric Radiation Measurement (ARM) Program: Programmatic Background and Design of the Cloud and Radiation Test Bed. AMS Vol. 75, No. 7, 1201:1221

Ware R. H., D. W. Fulker, S. A. Stein, D. N. Anderson, S. K. Avery, R. D. Clark, K. K. Droegemeier, J. P. Kuettner, J. B. Minster, S. Sorooshian, 2000: SuomiNet: A Real-Time National GPS Network for Atmospheric Research and Education, AMS Vol. 81, No. 4, 677:694

Related articles

• Page:
  ECV - Glaciers, Ice Sheets, Lakes, Soil Moisture
• Page:
  IV Version 1.2; SEC Version 2.0; GMB Version 2.0: Product User Guide Specification (PUGS)
• Page:
  IV Version 1.2; SEC Version 2.0; GMB Version 2.0: System Quality Assurance Document (SQAD)
• Page:
  IV Version 1.2; SEC Version 2.0; GMB Version 2.0: Product Quality Assurance Document (PQAD)
• Page:
  IV Version 1.2; SEC Version 2.0; GMB Version 2.0: Product Quality Assessment Report (PQAR)