Last modified on Dec 01, 2023 17:14

Contributors: W. Preimesberger (WP, TU Wien), W. Dorigo (WD, TU Wien), A. Dostalova (AD, EODC), R. Kidd (RK, EODC), T. Frederikse (TF, Planet Labs / Vandersat) 

Issued by: EODC GmbH/Alena Dostalova

Date: 06/09/2023

Ref: C3S2_312a_Lot4.WP2-FDDP-SM-v1_202212_SM_PQAR-v4_i1.2

Official reference number service contract: 2021/C3S2_312a_Lot4_EODC/SC1

History of modifications

List of datasets covered by this document

Related documents 

Acronyms

General definitions

Active (soil moisture) retrieval: the process of modeling soil moisture from radar (scatterometer and synthetic aperture radar) measurements. The measurand of active microwave remote sensing systems is called “backscatter”.

Accuracy: The closeness of agreement between a measured quantity value and a true quantity value of a measurand ((JCGM), 2008). The metrics used here to represent accuracy are correlation and unbiased Root Mean Square Difference (ubRMSD). These metrics are commonly used throughout the scientific community as measures of accuracy (Entekhabi et al., 2010).

Backscatter is the measurand of “active” microwave remote sensing systems (radar). As the energy pulses emitted by the radar hit the surface, a scattering effect occurs and part of the energy is reflected. The received energy is called “backscatter”, with rough surfaces producing stronger signals than smooth surfaces. It comprises reflections from the soil surface layer (“surface scatter”), vegetation (“volume scatter”), and interactions of the two. Under very dry soil conditions, structural features in deeper soil layers can act as volume scatterers (“subsurface scattering”).

Bias: “Bias is defined as an estimate of the systematic measurement error.” (GCOS-200) [RD5]

Breakthrough requirement: An Essential Climate Variable (ECV) requirement level set by Global Climate Observing System (GCOS) which “[…] if achieved, would result in a significant improvement for the targeted application […] at which specified uses within climate monitoring become possible. It may be appropriate to have different breakthrough values for different uses.” (GCOS-245) [RD6]

Brightness Temperature is the measurand of “passive“ microwave remote sensing systems (radiometers). Brightness temperature (in degree Kelvin) is a function of kinetic temperature and emissivity. Wet soils have a higher emissivity than dry soils and therefore a higher brightness temperature. Passive soil moisture retrieval uses this difference between kinetic temperature and brightness temperature, to model the amount of water available in the soil of the observed area while taking into account factors such as the water held by vegetation.

Dekad: the period or interval of 10 days

Error: “The term error refers to the deviation of a single measurement (estimate) from the true value of the quantity being measured (estimated), which is always unknown” (Gruber et al., 2020)

Fiducial Reference Measurements (FRM): Suite of independent, fully characterized, and traceable ground measurements that follow the guidelines outlined by the Group on Earth Observation (GEO) / Committee in Earth Observation Satellites (CEOS) Quality Assurance framework for Earth Observation (QA4EO)¹.

Key Performance Indicators (KPIs): A set of performance measures designed to rate the quality of satellite soil moisture observations. Based on suggestions from the Global Climate Observing System (GCOS), the Climate Modelling User Group (CMUG) and other community agreed standards.

Koeppen-Geiger Climate Classification: Global classification of regions based on their climates. Contains 5 main classes with multiple sub-classes: A (tropical), B (arid), C (temperate), D (continental), and E (polar) climates. In the context of Copernicus Climate Change Service (C3S) soil moisture the classification of Peel et al. (2007) is used.

Level 2 pre-processed (L2P): this is a designation of satellite data processing level. “Level 2” means geophysical variables derived from Level 1 source data on the same grid (typically the satellite swath projection). “Pre-processed” means ancillary data and metadata added following Group for High Resolution Sea Surface Temperature (GHRSST) Data Specification.

Level 3 /uncollated/collated/super-collated (L3U/L3C/L3S): this is a designation of satellite data processing level. “Level 3” indicates that the satellite data is a geophysical quantity (retrieval) that has been averaged where data are available to a regular grid in time and space. “Uncollated” means L2 data granules have been remapped to a regular latitude/longitude grid without combining observations from multiple source files. L3U files will typically be “sparse” corresponding to a single satellite orbit. “Collated” means observations from multiple images/orbits from a single instrument combined into a space-time grid. A typical L3C file may contain all the observations from a single instrument in a 24-hour period. “Super-collated” indicates that (for those periods where more than one satellite data stream delivering the geophysical quantity has been available) the data from more than one satellite have been gridded together into a single grid-cell estimate, where relevant.

Passive (soil moisture) retrieval: the process of modeling soil moisture from radiometer measurements. The measurand of passive microwave remote sensing is called “brightness temperature”). The retrieval model in the context of C3S soil moisture is generally the Land Parameter Retrieval Model.

Radiometer: Spaceborne radiometers are satellite-carried sensors that measure energy in the microwave domain emitted by the Earth. The amount of radiation emitted by an object in the microwave domain (~1-20 GHz). The observed quantity is called “brightness temperature” and depends on the kinetic temperature of an object and its emissivity. Due to the high emissivity of water compared to dry matter, radiometer measurements of the Earth’s surface contain information on the water content in the observed area.

Scatterometer: Spaceborne scatterometers are satellite-carried sensors that use microwave radars to measure the reflection or scattering effect produced by scanning a large area on the surface of the Earth. The initially submitted pulses of energy are reflected by the Earth’s surface depending on its geometrical and geophysical properties in the target area. The received energy is called “backscatter”. Soil moisture retrieval relies on the fact that wet soils have a higher reflectivity (and therefore backscatter) than dry soils due to the high dielectric constant of liquid water compared to dry matter.

Stability: “The change in bias over time” (GCOS-245) [RD6]. “Stability may be thought of as the extent to which the uncertainty of measurement remains constant with time. […] ‘Stability’ refer[s] to the maximum acceptable change in systematic error, usually per decade.” (GCOS-200) [RD5]

Target requirement: ideal requirement which would result in a considerable improvement for the target application.

Threshold requirement: minimum requirement to be met to ensure data are useful.

Uncertainty: “Satellite soil moisture retrievals […] usually contain considerable systematic errors which, especially for model calibration and refinement, provide better insight when estimated separately from random errors. Therefore, we use the term bias to refer to systematic errors only and the term uncertainty to refer to random errors only, specifically to their standard deviation (or variance)” (Gruber et al., 2020)

Scope of the document

The purpose of this document is to describe the results of the Quality Assessment (QA) for the soil moisture product developed by Vienna University of Technology (TU Wien), Earth Observation Data Centre (EODC), and Planet Labs/VanderSat for the Copernicus Climate Change Service (C3S) hosted by the European Centre for Medium Range Weather Forecasting (ECMWF). The product version assessed in this report is v202212.0.0, which was produced in May 2023.

Executive summary

The C3S soil moisture product suite provides PASSIVE, ACTIVE and COMBINED (passive + active) microwave soil moisture products on a daily, dekadal (10-days) and monthly basis. The data are provided in a regular 0.25-degree grid based on the World Geodetic System 1984 (WGS 84) reference system. The product is available globally between November 1978 and present-day (for PASSIVE and COMBINED) and between 1991 and present-day (for ACTIVE). For details about the products, we refer to the Product User Guide and Specification (PUGS) [RD1].

This document presents the results of QA activities that have been undertaken for the current Climate Data Record (CDR) product version v202212.0.0. The Intermediate Climate Data Records (ICDR) are currently not assessed. However, note that to achieve maximum consistency between CDR and ICDR, both products use the same Level 2 products (based on Near Real Time (NRT) data streams) and merging algorithms and thus have very similar quality characteristics.

Chapter 1 summarises the quality assessment methodology, described fully in the Product Quality Assurance Document (PQAD) [RD2]. The results described here are primarily for the COMBINED daily product, however, most assessments were also undertaken for the ACTIVE and PASSIVE products.

Chapter 2 contains QA results for (i) accuracy assessment, (ii) stability assessment, (iii) demonstration of uncertainty estimates, (iv) comparison to previous versions of the product, and (v) completeness/consistency assessment. The first two sections focus on demonstrating that the Key Performance Indicators (KPIs) set for the product are met. Note these KPIs take into account the Global Climate Observing System (GCOS) and user requirements for the product described in the Target Requirements and Gap Analysis Document (TRGAD) [RD4].

Accuracy Assessment: In general, there is a slight variability in the correlation between the datasets, with correlations ranging from 0.4 to above 0.8; depending on the conditions and the locations of the in situ stations used. The unbiased Root Mean Square Difference (ubRMSD), which can be directly taken as a measure of accuracy, is demonstrated to be below 0.10 m³ / m³ for all the different conditions analyzed. Therefore, the KPIs for accuracy have been met for in situ observations. The global comparison against the ERA5-Land (ECMWF Reanalysis v5) has shown that the KPI threshold is met in most areas, except for some areas with reduced data coverage in the sub-Arctic zone.

Stability Assessment: The stability of the C3S product has been assessed in terms of the change in accuracy (when compared to International Soil Moisture Network (ISMN) network measurements). The accuracy between the products (ubRMSD) has been calculated per year, as well as trends in the median yearly accuracy. The KPI threshold for stability of 0.05 m³ / m³ / y is met when assessed using this method for all tested locations.

A comparison to previous products has been provided. The assessment demonstrates that the correlation between the in situ and satellite-derived products of v202212.0.0 is slightly higher compared to previous versions. Improvements are found for some networks. Increases in the data coverage of all products are expected due to additional observations used in the most recent record.

The main algorithmic updates are:

• Data from Advanced Scatterometer (ASCAT)-C, FengYun (FY)-3C, and FY-3D are included for the first time in this version.
• Daytime observations are now included for all PASSIVE sensors except operational ones used to create the ICDR (Advanced Microwave Scanning Radiometer 2 (AMSR2), Soil Moisture and Ocean Salinity (SMOS), Soil Moisture Active Passive (SMAP) and Global Precipitation Measurement (GPM)).
• Land Parameter Retrieval Model (LPRM) v7.1 is used for all passive sensors (with the exception of the L-band sensors, where LPRMv6.2 is used). This includes improved brightness temperature calibration and day-time observations for all sensors
• Barren ground flag was introduced as an optional flag, corresponding to the bit 0b1000000
• An intra-annual rescaling methodology is applied to homogenize the individual sensor records.

The spatial and temporal coverage of the product has been presented in terms of the number of valid (unflagged) observations available. It is shown that the coverage is better in Europe, Southern Africa, and the contiguous United States (US) than in some other parts of the world.

Furthermore, a detailed assessment of the product has been undertaken, in particular for the ACTIVE and PASSIVE products as well as a detailed comparison against the previous product version. These assessments show that the dataset was generated correctly.

However, some (previously known) issues remain in the current version:

• ACTIVE wetting trends: Artificial wetting trends in Hydrological Satellite Application Facility (H-SAF) ASCAT Surface Soil Moisture (SSM) also affect the ACTIVE product of C3S Soil Moisture (SM) after 2007 (and, to some extent, the COMBINED product). These trends are already present in ASCAT observed backscatter time series. They are caused by changes in land cover that are not sufficiently accounted for during the soil moisture retrieval process and therefore translate into a gradual wetting bias in the data. The trends are currently corrected in an experimental version of ASCAT SSM. This issue will be corrected once the change correction is also introduced to the operational H-SAF SSM products used in C3S SM.
• Inconsistent long-term SM trends: Inconsistent long-term trends (1978/1991 – 2020) are found between the ACTIVE, PASSIVE and COMBINED products. While this can be partly related to differences in time range, data density, and scaling references of the products, it still reveals that there is a need to improve the long-term consistency of the product.

The following issues that were found in the previous version are resolved in the new release:

• ERA5 assessment: The assessment of previous C3S SM versions against the ERA5 dataset showed ubRMSD significantly higher than the KPI thresholds in some sub-Arctic areas. Due to recent updates in the flagging of frozen soils as well as a new model-independent temperature correction introduced in LPRM v7, this issue could be reduced significantly in v202212.0.0.

Product validation methodology

Validated products

This current document is applicable to the QA activities performed on the version of the CDR v202212.0.0 (produced in May 2023).

C3S soil moisture provides passive (named PASSIVE), active (ACTIVE), and passive + active (COMBINED) soil moisture CDRs on a daily, dekadal (10-days), and monthly basis (9 CDRs for each version). The periods, over which each sensor is used, are provided in Figure 1. The data are provided in a regular 0.25-degree grid based on the WGS 84 reference system. The product is available globally between November 1978 and present-day (for PASSIVE and COMBINED) and between 1991 and present-day (for ACTIVE). The product has been produced by TU Wien, Planet Labs / VanderSat, and EODC. The ASCAT SSM product mentioned in some parts of this report is developed and provided by H-SAF.

Figure 1: Sensors and merging periods for the C3S soil moisture product (ACTIVE, PASSIVE and COMBINED) version v202212.

The C3S soil moisture product is generated from a set of passive microwave radiometers and active microwave scatterometers (red and blue sensors respectively in Figure 1).

Radiometers include the Scanning Multichannel Microwave Radiometer (SMMR), Special Sensor Microwave Imager (SSM/I), Tropical Rainfall Measuring Mission (TRMM) Microwave Imager (TMI), Advanced Microwave Scanning Radiometer-Earth Observing System (AMSR-E), WindSat Spaceborne Polarimetric Microwave Radiometer (WindSat), Advanced Microwave Scanning Radiometer 2 (AMSR2), Soil Moisture and Ocean Salinity (SMOS), Soil Moisture Active Passive (SMAP), FengYun (FY) 3B/3C/3D and Global Precipitation Measurement (GPM) satellites.

Scatterometer observations are collected by the Active Microwave Instrument - Wind Scatterometer (AMI-WS) (onboard European Remote-Sensing Satellites (ERS)-1 and 2) and ASCAT (onboard Meteorological Operational (MetOp) satellites MetOp-A, MetOp-B and MetOp-C) sensors. The “ACTIVE product” and the “PASSIVE product” are created by fusing scatterometer and radiometer Level 2 soil moisture products respectively; the “COMBINED product” is created by fusing Level 2 soil moisture products from both sensor types. Data files are provided in NetCDF-4 classic format as daily, dekadal, and monthly images and comply with CF-1.8 conventions².

A detailed description of the product generation of C3S v202212.0.0 is provided in the Algorithm Theoretical Basis Document (ATBD) [RD3] with further information on the product given in the PUGS [RD1]. The underlying algorithm is based on that used in the generation of the publicly released European  Space Agency (ESA) Climate Change Initiative (CCI) version 7.1 which is described in relevant documents in Plummer et al. (2017), Wagner et al. (2012), Liu et al. (2012), and Dorigo et al. (2017). In addition, detailed provenance traceability information can be found in the metadata of the product.

Description of reference datasets

A combination of in situ and global reference datasets are utilized to assess the quality of the C3S soil moisture product. Datasets utilized are shortly described in this section with further details provided in the PQAD [RD2].

Compared to the previous version of this report, Global Land Data Assimilation System (GLDAS) Noah and ERA5 soil moisture products are no longer used as reference data sets for the validation of C3S SM. GLDAS is used in the generation of the COMBINED product and therefore not independent. ERA5-Land soil moisture fields are generally more accurate than ERA5 (Muñoz-Sabater et al. 2021).

International Soil Moisture Network (ISMN)

This ISMN³ is a database hosted by the German Federal Institute of Hydrology (BfG). It is a global collocation of available in situ soil moisture measurements from operational networks and validation campaigns. In situ data are collected from the individual network providers, harmonized, and made available to users (Dorigo et al. 2021). The data within the ISMN are subject to quality controls and provided with quality flags (Dorigo et al. 2013). The quality controls include an assessment against a possible range of important metrological variables which are applied equally to all datasets. The yearly temporal coverage and time series length vary largely between stations. Often there are multiple soil moisture sensors present at a station in similar depths.
A snapshot of the full ISMN archive was downloaded in January 2023. This snapshot contains 3554 soil moisture time series from sensors in the top 10 cm of soil (Figure 2).

Figure 2: ISMN stations (as of January 2023) and sensor count. Stations with 1 (top left), 2 (top right), 3-6 (bottom left), and >7 (bottom right) soil moisture sensors in 0-10 cm depth are shown separately.

As part of ESA's Fiducial Reference Measurements for Soil Moisture (FRM4SM) project https://project-frm4sm.geo.tuwien.ac.at/ (resource validated 03/07/2023), a quality index for the "representativeness" of each in situ time series for the satellite data at radiometer scale (~25 km) was produced from the time series length and the Triple Collocation Analysis (TCA) based Signal-to-Noise Ratio (SNR). Sufficiently long time series with a high SNR (>3 dB) at 95% confidence are classified as "very representative" or "representative". Applying this quality indicator to all ISMN time series results in a subset of 1083 high-quality sensor time series. Their location is shown in Figure 3. The so-found subset forms the reference for the evaluation of the C3S SM.

Figure 3: Subset of the ISMN database to include "very representative" and "representative" measurements. Based on the FRM4SM quality indicator. Note that points in the graphic often overlap as most networks consist of stations that are spatially close together (clusters).

ISMN stations are distributed unevenly globally and often appear in clusters. Most measurements are taken within the continental United States and Europe. Validation results are therefore only representative of the environmental regimes covered by these stations (temperate climates) and mostly exclude (sub)equatorial and (sub)polar regions, highly organic soils as well as deserts.

Pre-processing

To calculate the metrics for each assessment, the settings summarized in Table 1 are used. The datasets are spatially and temporally matched, filtered for high-quality observations, and the C3S data are scaled to the ISMN data using mean/standard deviation scaling. The metrics are then calculated using all available observations in the period 1978-11-01 to 2022-12-31.

Table 1: Settings used in the assessment of the daily C3S soil moisture product against the ISMN

ERA5-Land

ERA5-Land (Muñoz-Sabater et al. 2021), produced by ECMWF, is available from 1950 to present with a few days delay. It provides surface variables with an increased spatial resolution compared to ERA5 (Hersbach et al. 2020). Soil Moisture in ERA5-Land is available in 1-hour intervals on a ~9km resolution and without gaps. ERA5-Land provides various (land) variables, such as soil moisture or soil temperature. Both are available for four soil layers (0-7, 7-28, 28-100 and 100-289 cm). Here we use ERA5-Land data from 1981 to 2022 extracted for hours 0, 6, 12, and 18 of each day. Data for the top soil layer (0-7 cm) is used for comparison to the C3S SM satellite products.

ESA CCI SM

The CCI project was initiated in 2009 by ESA in response to the United Nations Framework Convention on Climate Change (UNFCCC) and GCOS needs for Essential Climate Variable (ECV) databases (Plummer et al. 2017). In 2012, ESA released the first multi-decadal, global satellite-observed soil moisture dataset, named ESA CCI SM, combining various single-sensor active and passive microwave soil moisture products (Dorigo et al. 2017).

The C3S product at version v202212 is based (scientifically, algorithmically) on version v7.1 of ESA CCI SM. There are however small differences in terms of integrated data streams due to the operational production of C3S ICDRs: C3S SM uses different ASCAT data streams that provide observations in NRT. Daytime observations are not used for any of the operational passive sensors as the current daytime retrieval algorithm is not compatible with operational products. The LPRM data used in ESA CCI SM and C3S SM is not exactly the same. C3S uses a reprocessed version of all sensors with some changes and bug fixes applied after the production of ESA CCI SM.

Description of product validation methodology

Method Overview

The methodology used in the assessment of the soil moisture product is described in the PQAD [RD2]. The methodology, including details on the validated products and the validating datasets, are briefly described here in the relevant sections. A summary of the validation results for the previous dataset version (v202012) may be found in the relevant PQAR document [RD7]. All box plots in the following chapters indicate the median, interquartile range, 1.5 times interquartile range of metrics found between the satellite and reference data products.

The quality assessment includes the following:

1. Accuracy assessment against in situ observations from the ISMN (Section 2.1),
2. Accuracy assessment against ERA5-Land reanalysis (Section 2.2),
3. Stability analysis through monitoring of accuracy trends and dataset statistics (Section 2.3),
4. Assessment of the spatial and temporal completeness and consistency of the product (Section 2.4),
5. Analysis of time series at selected locations (Section 2.5), and,
6. Analysis of the uncertainty information provided with the dataset (Section 2.6).

Additional information is also provided in the Annexes of the report:

1. Annex A: The main body of the report considers mainly validation of the COMBINED product CDR. Therefore, a separate Annex provides information on the validation results of the ACTIVE and PASSIVE products, summarising the key findings from these activities.
2. Annex B: The main body of the report provides some comparison between the current C3S version (v202212) and the previous version (v202012). Annex B provides a more complete analysis to demonstrate that the data have been produced correctly and show any differences between the products.
3. Annex C: The main body of the report refers to the daily images provided through CDS. This section evaluates the aggregated product (monthly and 10-day /dekadal), which can also be downloaded.

Quality Assurance for Soil Moisture

Some analysis presented in this report has been undertaken using software specifically designed for the C3S project. However, in parallel, an online validation service - Quality Assurance for Soil Moisture (QA4SM)⁶ - has been developed, which undertakes similar tasks and is used for most of the global validations.

The QA4SM service is an online validation service, which allows the traceable validation of state-of-the-art satellite-derived soil moisture products (C3S SM, SMAP, SMOS, H-SAF ASCAT SSM, ESA CCI SM, etc.) as well as any user uploaded data sets. The comparisons can be carried out against in situ and model/reanalysis reference data and other satellite products. CDR v202212 of C3S SM is available online.

Validation results produced using the QA4SM service have a permalink assigned. This redirects users to archived, traceable, citable results on the Zenodo platform⁷. The relevant links are given in each chapter.

Validation results

Accuracy – Comparison against ISMN

Introduction

The C3S SM COMBINED dataset has been compared to the ISMN dataset and the correlations and ubRMSD between the two datasets calculated. A mean and standard-deviation matching is applied to bring satellite and in situ observations into the same value range.

In addition to an overall comparison processed using the QA4SM service (Section 2.1.2), the comparison is also undertaken for different attributes of the soil moisture data (provided as metadata within the ISMN dataset). These are: the sensor depth (Section 2.1.2.1), soil texture (Section 2.1.2.2), Köppen-Geiger climate classes (Section 2.1.2.3), and land cover (Section 2.1.2.4). For further information on the origin of these attributes, see Dorigo et al. (2011).

The aim of assessing the accuracy for different attributes is to demonstrate the performance of the product under different conditions and demonstrate that KPIs DX.1 (where X is equal to 1 – 6) are met. See the PQAD [RD2], for the list of KPIs; repeated in this document in Table 4.

The focus here is on the COMBINED product as this is the recommended product for most applications and designed to theoretically outperform the ACTIVE and PASSIVE products.

Validation runs from this chapter are traceable. The following validation runs are available on QA4SM and published on Zenodo:

• Validation run: C3S SM COMBINED v202212 vs v202012 vs ISMN Fiducial Reference Measurements (FRMs) (absolute) 0-10 cm
  • https://qa4sm.eu/ui/validation-result/240161ef-60c6-4224-805d-5361d36b2a82
  • DOI: https://doi.org/10.5281/zenodo.8100137
• Validation run: C3S SM COMBINED v202212 vs v202012 vs ISMN FRMs (anomalies) 0-10 cm
  • https://qa4sm.eu/ui/validation-result/ba00b64b-3b61-4426-9b07-29df7e8e3620
  • DOI: https://doi.org/10.5281/zenodo.8100151
• Validation run: C3S SM COMBINED v202212 vs v202012 vs ISMN FRMs (absolute) 0-5 cm
  • https://qa4sm.eu/ui/validation-result/44c5ab62-c46e-4ebb-b99d-0bbf33e8b17f
  • DOI: https://doi.org/10.5281/zenodo.8101045
• Validation run: C3S SM COMBINED v202212 vs v202012 vs ISMN FRMs (absolute) 5-10 cm
  • https://qa4sm.eu/ui/validation-result/6a33bb0a-7744-4950-8478-d55b234d4b29
  • DOI: https://doi.org/10.5281/zenodo.8101051
• Validation run: C3S SM COMBINED v202212 vs v202012 vs ISMN FRMs (anomalies) 0-5 cm
  • https://qa4sm.eu/ui/validation-result/388acf49-4cd6-4b32-9abb-e0f343c48cfd
  • DOI: https://doi.org/10.5281/zenodo.8101053
• Validation run: C3S SM COMBINED v202212 vs v202012 vs ISMN FRMs (anomalies) 5-10 cm
  • https://qa4sm.eu/ui/validation-result/5219f9fd-d309-4405-befb-cf829f695bee
  • DOI: https://doi.org/10.5281/zenodo.8101062
• Validation run: C3S SM COMBINED all versions (v202212)+ERA5-Land vs ISMN FRMs 0-10 cm
  • https://qa4sm.eu/ui/validation-result/2f865302-5e35-4c28-8549-1cc7e86f38cd
  • DOI: https://doi.org/10.5281/zenodo.8101626 ⁸

Validation results

The comparison of C3S v202212 has been processed using the QA4SM service against ISMN FRMs at v20230110. The global map in Figure 4 shows the correlation (Pearson's) for each ISMN station to the nearest C3S grid cell; the same is shown for ubRMSD in Figure 5. These figures show the expected spatial patterns, with correlations between 0.6-0.7 and ubRMSD between 0.04-0.06 m³/m³ seen at most ISMN locations (as required below the minimum ubRMSD KPI threshold of 0.1 m³/m³). COMBINED v202212 performs slightly better in this comparison than v202012.

The same analysis is repeated for soil moisture anomalies from both data sets. Available observations in the period between 1990 and 2020 are averaged and subsequently smoothed with a 30-day moving window to create the climatological reference. Anomaly correlations indicate how well individual precipitation or drought events at the in situ stations are captured by the ~25 km satellite grid cells.

The ubRMSD is generally higher, and the correlation is lower respectively, compare to the absolute values (Figure 6). This is expected, as the well-represented seasonal signal component in both time series is no longer taken into account. However, same as for the absolute values, differences between the two C3S SM versions are small. Some improvements are found for the new version.

Figure 4: Correlation (Pearson's) between C3S v202212 COMBINED and ISMN v20230110 for soil depths of 0 – 10 cm (top) and comparison with C3S v202012 COMBINED (bottom). Produced via https://qa4sm.eu. White boxes contain the computed validation scores for all ISMN sensors (N),coloured boxes indicate the lower and upper limits of the 95% confidence interval at the same locations.

Figure 5: ubRMSD between C3S v202212 COMBINED and ISMN v20230110 for soil depths of 0 – 10 cm (top) and comparison with C3S v202012 COMBINED (bottom). Produced via https://qa4sm.eu. White boxes contain the computed validation scores for all ISMN sensors (N),coloured boxes indicate the lower and upper limits of the 95% confidence interval at the same locations.

Figure 6: ubRMSD (top) and correlation (bottom) between anomalies from C3S v202212 and v202012 COMBINED and ISMN v20230110 sensors from 0 – 10 cm. Produced via https://qa4sm.eu. White boxes contain the computed validation scores for all ISMN sensors (N),coloured boxes indicate the lower and upper limits of the 95% confidence interval at the same locations.

Stratification - Soil Depth

The correlation (Figure 7) and ubRMSD (Figure 8) between the last two versions of C3S SM COMBINED and the in situ datasets are presented for two different surface soil moisture depths (0-5 cm and 5-10 cm) using the absolute values as well as anomalies.

The deeper sensors have a lower correlation and higher ubRMSD with C3S SM satellite observations than the sensors at shallower depths. This is expected as the product represents the first few centimeters of the soil surface (approx.). However, it may also be attributed to the decreasing errors associated with the ISMN data at these deeper depths - noting that the relationship seems to break at depths > 1 m (Gruber et al. 2013).

Figure 7: Correlation between C3S SM v202212 COMBINED and ISMN FRMs for soil depths of 0 – 5 cm and 5 – 10 cm for absolute values (top) and anomalies (bottom). Wide boxes contain the computed validation scores for all ISMN sensors (N), narrow boxes indicate the lower and upper limits of the 95% confidence interval at the same locations.

Figure 8: ubRMSD between C3S SM v202212 COMBINED and ISMN FRMs for soil depths of 0 – 5 cm and 5 – 10 cm for absolute values (top) and anomalies (bottom). Wide boxes contain the computed validation scores for all ISMN sensors (N), narrow boxes indicate the lower and upper limits of the 95% confidence interval at the same locations.

Stratification - Soil Texture

The correlation and ubRMSD between the C3S dataset and the in situ datasets are presented in Figure 10 and Figure 11 for the different soil textures (fine, medium, and coarse; stratification provided from the ISMN dataset (Dorigo et al. 2011) and shown in Figure 9).

Figure 9: Soil texture classification used for the ISMN comparison. The percentage clay, silt, and sand are taken from the ISMN metadata which in turn is retrieved from the Harmonised World Soil Database (HWSD).

The product appears to perform best for medium or fine texture soils in terms of correlation and ubRMSD (for both depth layers). The spread in ubRMSD for fine soil textures is higher than for medium textures. Notice that the number of available sensors between soil texture classes changes significantly for different depths. While for most sensors in the top four centimeters "Coarse granularity" is assigned, "Medium granularity" is the predominant class for sensors below four centimeters. Compared to the previous version, correlation increased especially with sensors in 5-10 cm depths (all soil classes).

Figure 10: Pearson's R between absolute values of C3S SM COMBINED and ISMN for different soil texture classifications and depths.

 

Figure 11: ubRMSD between absolute values of C3S SM COMBINED and ISMN for different soil texture classifications and depths.

Differences between the two versions are smaller when comparing anomaly correlations and ubRMSD (Figure 12 and Figure 13). In some cases (especially for sensors at 5 cm depth), v202012 slightly outperforms v202212.

Figure 12: Pearson's R between anomaly values of C3S SM COMBINED and ISMN for different soil texture classifications and depths.

Figure 13: ubRMSD between anomaly values of C3S SM COMBINED and ISMN for different soil texture classifications and depths.

Stratification - Köppen-Geiger climate classes

The correlation and ubRMSD between the C3S datasets and the in situ datasets are presented for different Köppen-Geiger classes (summarised as "Tropical", "Arid", "Temperate", "Continental" and "Polar") a global map of which is shown in Figure 14 (stratification provided from the ISMN metadata - see Dorigo et al. (2011)).

The correlation (absolute values in Figure 15, anomaly values in Figure 17) and ubRMSD (Figure 16 and Figure 18) between the in situ measurements and the C3S COMBINED product varies across different Köppen-Geiger classes. Comparing the performance for stations in "Arid" and "Continental" and "Temperate" climates (which are the three classes with a relatively large number of stations available), "Temperate" climate showing the highest correlation and second lowest ubRMSD with the satellite data in this comparison.

Notably "Tropical" and "Polar" perform surprisingly well in this comparison (compared to previous studies). This could either be due to the relatively low number of ISMN stations in these areas, but might also be related to the previously discussed selection of ISMN FRM sites (Chapter 1.2.1). Satellite products in these areas might perform better than previously assumed when taking into account the representativeness of in situ sites (land cover homogeneity, length of time series, etc.), and that the selection of representative reference data in these areas is more important than for temperate or arid zones to get a meaningful quality assessment.

The graphs show that in this case, the median ubRMSD values for all of the different climate classes are under the 0.1 m³/m³ minimum threshold set out in the KPIs; therefore the KPIs are achieved. The target GCOS threshold of 0.04 m³/m³ is approached and only reached in some cases.

Figure 14: Köppen-Geiger classes. The classes used in this assessment are summarised as "Tropical", "Arid" "Temperate", "Continental, "Polar". The figure is taken from http://hanschen.org/koppen.

Figure 15: Correlation (Pearson's) between absolute values of C3S SM COMBINED and ISMN for different climate classes in 0-10 cm depth.

 

Figure 16: ubRMSD between absolute values of C3S SM COMBINED and ISMN for different climate classes in 0-10 cm depth.

 


Figure 17: Correlation (Pearson's) between anomaly values of C3S SM COMBINED and ISMN for different climate classes in 0-10 cm depth.

 

Figure 18: ubRMSD between anomaly values of C3S SM COMBINED and ISMN for different climate classes in 0-10 cm depth.

Stratification - Landcover classes

The correlation and ubRMSD between the C3S dataset and the in situ datasets are aggregated for five different land cover classes: i) "Cropland", (ii) "Grassland", (iii) "TreeCover", (iv) "Urban Areas", (v) "Other"; stratification provided from the ISMN dataset (Dorigo et al. 2011).

Stations with land cover class "Grassland" assigned show the highest correlation (and lowest ubRMSD) with the satellite products among classes with a sufficient number of stations. Points with "TreeCover" on the other hand perform worst for both metrics between absolute values (Figure 19 and Figure 20) as well as anomalies (Figure 21 and Figure 22). The new C3S COMBINED product outperforms the previous version in all relevant cases.

The ubRMSD can be taken as a measure of accuracy and the KPIs specifically state the acceptable accuracy level of the product is between 0.01 and 0.1 m³/m³ for different land cover types. The worst case ubRMSD shown here is under 0.1 m³/m³ (median and Interquartile Range (IQR)) for tree cover. Therefore, overall the minimum KPIs are met. The GCOS target of 0.04 is m³/m³ is approached and only reached in some cases.

Figure 19: Correlation (Pearson's) between absolute values of C3S SM COMBINED and ISMN for different land cover classes in 0-10 cm depth.

 

Figure 20: ubRMSD between absolute values of C3S SM COMBINED and ISMN for different land cover classes in 0-10 cm depth.

 

Figure 21: Correlation (Pearson's) between anomaly values of C3S SM COMBINED and ISMN for different land cover classes in 0-10 cm depth.

 

Figure 22: ubRMSD between anomaly values of C3S SM COMBINED and ISMN for different land cover classes in 0-10 cm depth.h3.

Comparison to all previous versions

A comparison of the correlation between satellite SM and in situ observations is provided in Figure 23 and Figure 24 for different versions of the C3S product. Absolute soil moisture values are used in this comparison. Only time stamps with valid observations in all data sets are used. No bias correction (scaling) is applied. ERA5-Land temperature is used to remove all observations when soil temperate is below 0° C. It can be seen that the overall correlation and ubRMSD between the in situ and satellite-derived soil moisture datasets improve with the release of new product versions over time.

Figure 23: Correlation between the last four C3S SM versions, ERA5-Land, and ISMN FRMs. White boxes contain the computed validation scores for all ISMN sensors (N),coloured boxes indicate the lower and upper limits of the 95% confidence interval at the same locations.


Figure 24: ubRMSD between the last four C3S SM versions, ERA5-Land, and ISMN FRMs. White boxes contain the computed validation scores for all ISMN sensors (N),coloured boxes indicate the lower and upper limits of the 95% confidence interval at the same locations.

In addition, ISMN is compared to v202212 as well as the three C3S SM versions before that, and ERA5-Land in 6 different sub-periods with a length of 4 years each i) 1996-1999, (ii) 2000-2003, (iii) 2004-2007, (iv) 2008-2011, (v) 2012-2015, (vi) 2016-2019. An increase in R of all C3S SM versions over time is found. Higher correlations with ISMN are found for v202212 than for previous versions. ERA5-Land shows overall better correspondence with the in situ data than the satellite products, especially in earlier periods. Relative differences between satellite and reanalysis products decrease significantly after 2015. This is likely due to the increased observations available as input to the C3S satellite products within this period. Especially the period after 2015 is positively affected by the addition of SMAP and shows higher correlations than the previous version.

Figure 25: Correlation between ISMN FRMs (0-10 cm) and C3S satellite products and ERA5-Land for different periods.

In general, the products were deemed to agree well with in situ observations, but lack behind the performance of those obtained for Land Surface Model (LSM) simulation integrating observed precipitation, such as ERA5-Land. This may be due to the discrepancy between the installation depth of the in situ probes (typically 5 cm) and the typical depth of ~2 cm represented by the satellite-derived datasets (Dorigo et al. 2017).

To demonstrate the differences between the active, passive, and combined products for the previous (v202012) and the current version (v202212), a summary of the comparisons of the dataset to the ISMN data is shown in Table 2. Overall, performance has improved for PASSIVE and COMBINED products over both the latest merging period and the complete data period. Performance has remained the same for the ACTIVE product.

Table 2: Results of comparison against ISMN (0-5 cm) for different C3S dataset versions (median values). Note, that the ubRMSD for ACTIVE product is provided in percentage of total saturation while for PASSIVE and COMBINED products are provided in m³/m³.

Accuracy – Comparison against ERA5-Land reanalysis

C3S COMBINED v202212 has been compared against ERA5-Land top layer Soil Moisture (from 2001-01-01 to 2019-04-30). Metrics are computed from absolute soil moisture values as well as anomaly values (the full period is used to derive the climatological reference). Validation runs from this chapter are traceable. The following validation runs are available on QA4SM and published on Zenodo:

• Validation run: C3S SM COMBINED v202212 vs v202012 vs ERA5-Land SWVL1 (absolute)
  • https://qa4sm.eu/ui/validation-result/80beb102-f5f4-4b1a-94f4-57008df9e322
  • DOI: https://doi.org/10.5281/zenodo.8146596
• Validation of C3S SM COMBINED v202212 vs v202012 vs ERA5-Land SWVL1 (anomalies)
  • https://qa4sm.eu/ui/validation-result/aaf66b58-ce30-4754-9349-8c48dcf81595 ⁹
  • DOI: https://doi.org/10.5281/zenodo.8146583

Comparison of absolute values

C3S COMBINED v202212 has been compared against ERA5-Land top layer Soil Moisture (from 2001-01-01 to 2019-04-30). Correlation and ubRMSD are shown in Figure 26 and Figure 27 respectively.

Figure 26 shows expected spatial patterns in the correlation coefficient (Pearson's) between ERA5-Land and C3S SM v202212. There is a high positive correlation between the products in most temperate zones, with lower correlations being most prevalent in the northern, boreal regions. The deserts show little to no positive correlation. Comparison to the previous C3S SM version indicates a higher agreement between the reanalysis and satellite produces with the new version. This is mainly due to a better agreement in subarctic regions in the new version due to the introduced cross-flagging and updated LPRM retrieval algorithm. The same can be seen in terms of ubRMSD in Figure 27. ubRMSD is generally low (~0.05 m³/m³), with a significant portion being below the 0.1 m³/m³ threshold required in the KPIs (see Section 4) and in some areas even below 0.04 m³/m³.

Figure 26: Correlation (Pearson's) between absolute soil moisture values of the C3S SM v202212 COMBINED product and ERA5-Land top level soil water content (top) and global comparison with the previous C3S SM version (bottom). White boxes contain the computed validation scores for all ISMN sensors (N), coloured boxes indicate the lower and upper limits of the 95% confidence interval at the same locations.

Figure 27: ubRMSD between absolute soil moisture values of the C3S SM v202212 COMBINED product and ERA5-Land top-level soil water content (top) and global comparison with the previous C3S SM version (bottom). White boxes contain the computed validation scores for all ISMN sensors (N), coloured boxes indicate the lower and upper limits of the 95% confidence interval at the same locations.

To compare the results for the current and previous version spatially (v202212 vs v202012), differences in correlation and ubRMSD are computed. Improvements in correlation and reduction in ubRMSD of absolute values are shown in blue in Figure 28. In general, the new version shows higher correlations with ERA5-Land than v202012, especially in high latitudes, deserts, and some distinct spots in South-East Asia. However, due to the newly introduced intra-annual bias correction, with modeled soil moisture from GLDAS Noah as the scaling reference, irrigation signals in some could be lost. Assuming that these features are also not represented in ERA5-Land, this could lead to a higher correlation than in the previous version. However, removing irrigation signals from the satellite data is not intended, and should be investigated further. The same comparisons for anomalies are shown in Figure 29.

In some areas (Europe, Arabia) a slight degradation is visible when comparing all values after 2001.

Figure 28: Change in absolute value correlations (top) and ubRMSD (bottom) wrt. ERA5-Land swvl1 between C3S SM COMBINED v202212 and v202012. Blue areas indicate better agreement between the new satellite dataset version and reanalysis.

Figure 29: Change in anomaly correlations (top) and ubRMSD (bottom) wrt. ERA5-Land swvl1 between C3S SM COMBINED v202212 and v202012. Blue areas indicate better agreement between the new satellite dataset version and reanalysis.

Stability – Trend monitoring

Methods for monitoring the stability of the dataset are still under development. Here, preliminary results are presented to demonstrate the methods developed so far along with a discussion of how they will be developed in the future.

Accuracy evolution

To assess the evolution of the C3S SM dataset quality over time, a preliminary analysis is performed of the evolution of accuracy over the 2004 to 2021 period, using ISMN as reference measurements. In this case not only FRMs are used, but all available ISMN sensors between 0 and 5 cm depth. A bias correction is applied by scaling the satellite data to the insitu observations via mean and standard deviation matching.

Figure 30 shows the evolution of Pearson's R for different land cover types. It shows that the stability of C3S SM (COMBINED) varies, depending on the land cover type. An important factor to consider in this comparison is the number of ISMN stations available each year, which is represented by the small number below the boxes. Notably, there are more ISMN stations available over time. The product appears to be most stable for "Grasslands", while for other classes there is visible variation in the product. For most classes, the highest R is found around the year 2016.

Similar observations can be made in terms of ubRMSD (Figure 31). Here, the expected widespread of error for the "Urban areas" class is also visible. This is probably caused by the spatial resolution of C3S SM, where soil moisture networks close to densely populated areas are less representative of the whole C3S SM cell as satellite SM in these areas can be affected by landcover changes (city growth) and Radio Frequency Interference (RFI).

The COMBINED product of C3S SM is below the ubRMSD KPI threshold of 0.1 m³/m³ in terms of median and IQR. For stations with landcover class "Tree Cover" a few points are found which exceed this threshold.

Figure 30: Accuracy evolution of C3S v202212 COMBINED between 2004 and 2021 in terms of Pearson's R based on land cover classes. The numbers at the bottom indicate the number of ISMN stations used in the comparison.

Figure 31: Accuracy evolution of C3S v202212 COMBINED between 2004 and 2021 in terms of ubRMSD based on land cover classes. The numbers at the bottom indicate the number of ISMN stations used in the comparison.

A similar analysis was performed based on four groups of climate classes (described in Section 2.1.2.3). Figure 32 and Figure 33 show that the performance is most stable for the "Cf / Df" (Temperate-Without Dry Season / Cold Without Dry Season) classes. A larger variation is found for the remaining classes.

The COMBINED product of C3S SM is below the ubRMSD KPI threshold of 0.1 m³/m³ in terms of median and IQR. For climate classes "C / D" (and their subclasses) single points are found outside of this threshold.

Figure 32: Accuracy evolution of C3S v202212 COMBINED between 2004 and 2021 in terms of Pearson's R based on climate classes. The numbers at the bottom indicate the number of ISMN stations used in the comparison.

Figure 33: Accuracy evolution of C3S v202212 COMBINED between 2004 and 2021 in terms of ubRMSD based on land cover classes. The numbers at the bottom indicate the number of ISMN stations used in the comparison.

The presented accuracy evolution results are the baseline for the estimation of stability trends in the dataset. Theil-Sen slopes were fitted to the median annual ubRMSD estimates. The so-found slope is supposedly representative of the long-term stability of the product. Note that this approach is currently under development and might change for future validation activities. Figure 34 shows the distribution of trends for different classes. At the tested locations for all land cover and climate classes, the change in accuracy metrics over time is very small, which indicates a stable SM product.

Figure 34: Distribution of trends in ubRMSD in C3S SM v202212 COMBINED for different land cover (panelsa to d) and climate (panels f to i) classes, tested against ISMN stations, where at least 3 years of accuracy evolution assessment was possible.

Spatial and temporal completeness

It is important to consider the spatial and temporal completeness and consistency of the product as these can be the key deciding factors for the users in terms of whether the product is suitable for their application.

The spatial and temporal coverage of the product is presented (shown in Figure 35 and Figure 36) in terms of the number of observations available. The figures show that the coverage is better in Europe, South Africa, and the contiguous US than in some other parts of the world, as well as the improvement in the availability of data post-2015 as SMAP became available (see Figure 1 for further details on sensor periods). This is expected for the product due to the orbital paths of the satellites resulting in higher coverage in equatorial regions. The reduced coverage in boreal and tropical regions is expected due to the presence of frozen soils for long periods and high Vegetation Optical Depth (VOD) respectively.

Figure 35: Fractional coverage of the C3S SM v202212 COMBINED product for the full period (top) and after 2015-04-01 (bottom). Expressed as the total number of daily observations per period divided by the number of days spanning that period.

 

Figure 36: Fraction of days per month with valid observations of SM for each latitude and time period for the v202212 COMBINED product. The coverage is expressed by month and relative to the total number of land points per latitude.

Time series analysis

Analysis of time series from a small number of locations provides insight into the behavior of the product for different climate and land cover types. Five points have been chosen for which ISMN in situ data are available (and were used in the above assessment). Details of the points are provided in Table 3 and are shown on a global map in Figure 37.

Table 3: Details of locations chosen for time series analysis.

Note: all are classified as having ‘medium’ soil texture.

Figure 37: Locations of the points used in the time series comparison (points are given at the C3S GPI location).

The time series (temporally aggregated per month) for the individual locations for the ACTIVE, PASSIVE and COMBINED products are given in Figure 38. In general, the time series appear to follow expected seasonal cycles at each location, i.e. winters are wetter and summers drier and, in the case of Grid point index (GPI) 890047 (which is located in Alaska), there are gaps in the data where the location is covered by snow each winter. For some locations (e.g. GPI 733335 in Figure 38) a sudden drop in SM is visible, corresponding to the introduction date of SMOS, however, this is less pronounced in COMBINED and therefore most likely related to the scaling applied in generating the PASSIVE product. GPI 316669 shows a steep increase in ACTIVE over the last 3 years, which is most likely related to land cover changes and/or insufficient inter-calibration of ASCAT-C (compare Section A.3).

Figure 38: Time series of soil moisture (COMBINED/PASSIVE in [m³/m³], ACTIVE in [% sat. / 100])  comparison for the COMBINED, ACTIVE and PASSIVE products of C3S v202212 for the GPIs and land cover types stated for each plot (compared with Figure 30). Note: here, the ACTIVE product is divided by 100 to allow it to be plotted on the same axis as the other products.

Uncertainty analysis

The algorithm used to develop the C3S soil moisture product utilizes triple collocation analysis to generate weightings for the combination of different soil moisture observations (Gruber et al. 2017). The SNR calculated as part of the triple collocation process is used to assign weights for merging. In combination with error propagation techniques, a per-pixel uncertainty is provided within the C3S soil moisture product in the "sm_uncertainty" field.

Figure 39 shows the SNR of ASCAT, and Figure 40 shows the same for new passive sensors, that were introduced to C3S SM first in version v202212. Note that, compare to ASCAT, all passive sensors have a higher (gap-filled) SNR in deserts and lower SNR for areas with dense vegetation. Daytime retrievals are generally performing worse than nighttime retrieval in LPRM as can be seen from the SNR maps in the right column of Figure 40. The theoretical issue of non-existing thermal equilibrium for midday observations present within the vegetation and soil surface leads to an increase in overall noise within the daytime datasets. Day-time retrievals are therefore assigned a lower merging weight compared to night-time data, and are primarily used to fill data gaps. No day-time retrievals are used for any of the passive sensors forming the ICDRs (SMAP, SMOS, AMSR2, and GPM). At the moment, day-time retrievals are not applicable for near-real-time production, and therefore also excluded from the CDR for consistency.

Figure 39: Signal-to-Noise Ratio for ASCAT (active sensor) derived from Triple Collocation Analysis. The SNR is used to weigh sensors for merging.

Figure 40: Signal-to-Noise Ratio of newly added passive sensors in C3S SM v202212. Derived from Triple Collocation Analysis. The SNR is used to weigh sensors for merging.

The evolution of the "sm_uncertainty" field per latitude over the C3S v202212 product period is shown in Figure 41.

It is expected that the uncertainty associated with the product reduces over time. This indicates that in sensor periods closer to the present, the original products are much closer together in absolute values than in the sensor periods nearer to the start of the C3S product. Throughout the product period, the uncertainties are always higher at latitudes where there is higher vegetation cover, for example at 10 degrees south. This is expected as soil moisture is harder to retrieve in these areas, and there is higher variance in the product where this is the case.

Figure 41: Monthly averages of the uncertainty variable associated with the C3S SM v202212 COMBINED product per latitude over time.

Application(s) specific assessments

European State of the Climate 2020

The C3S SM PASSIVE data are used in the "European State of the Climate 2022" report produced by ECMWF¹⁰. In the report, the C3S SM v202012 PASSIVE SM anomaly data (Figure 42, right) are compared against ERA5 SM anomalies (Figure 42, left). The anomalies shown match well with ERA5, although the satellite SM value range seems to be larger than for reanalysis SM, especially in the northern parts of Europe. However, anomalies may be affected by differences in the availability of data under frozen soil conditions (ERA5 is gap-free).

Figure 42: Seasonal soil moisture anomaly for 2022 in ERA5 (left) and C3S v202012 PASSIVE (right). From the "European State of the Climate 2022" report.

In addition, root-zone-soil-moisture was derived from the C3S SM surface product (Pasik et al. 2023). Z-scores are computed to express anomalies relative to the expected climatological variability and therefore indicate drought conditions in terms of standard deviations from the expected mean. Figure 43 shows the so-derived classification for June 2022, when the drought in western Europe peaked in terms of affected area.

Figure 43: Soil moisture anomaly z-scores for 0-100 cm root-zone-soil-moisture (RZSM) derived from C3S SM for June 2022.

Compliance with user requirements

The requirements for the C3S soil moisture product are a set of KPIs defined from consideration of user and GCOS requirements. The KPIs are shown in Table 4.

Table 4: Key Performance Indicators (KPIs) for the C3S Soil Moisture Product

An independent accuracy assessment has been conducted comparing the product against in situ observations with different conditions taken into account, i.e. soil depth, land cover etc. In general, there is variability in the correlation between the datasets, with correlations ranging from 0.4 to above 0.8, depending on the conditions and the locations of the in situ stations used. The ubRMSD, which can be directly taken as a measure of accuracy, is demonstrated to be below 0.10 m³ / m³ for all the different conditions analyzed, including land cover type (currently cropland, grassland, and tree cover are considered). Therefore, the minimum threshold KPI for the C3S product for accuracy has been met globally. However, the GCOS target requirement of 0.04 m³/m³ is only met under certain land cover (grassland) and climate (semi-arid) conditions. The median ubRMSD for most comparisons was found ~0.05 m³/m³. Notice, however, that the use of in situ data for satellite validation is complicated by the presence of representativeness errors, which inflates the actual errors. The effect of representativeness will be addressed in future quality assurance activities, as a finer categorization of the product skill according to land cover type or vegetation cover.
A comparison against the ERA5-Land reanalysis has also been undertaken to provide a wider global view of the product. The comparison against ERA5-Land has shown expected results, with the minimum KPI threshold of 0.1 m³ / m³ being met in almost all regions (the exception being a few areas with high topographic complexity). The target of 0.04 m³ / m³ is only reached in some areas. Similar to the comparison to in-situ observations, the median ubRMSD is found at around 0.05 m³ / m³ in most comparisons.

The stability of the C3S product has been assessed in terms of the change in yearly accuracy compared to ISMN stations for all years after 2004. The accuracy between the products (ubRMSD) has been calculated per year and the trends in the accuracy were also analyzed. The minimum KPI threshold for stability of 0.05 m³ / m³ / y is met when assessed using this method for all tested stations. The target of 0.01 m³ / m³ / y is met in most cases, with vegetation (changes) leading to lower product stability in some locations. More methods to assess SM stability are currently under investigation and will be presented in future evaluation studies.

Annex A: Outcomes of the ACTIVE and PASSIVE quality control

A.1 Introduction

The ACTIVE and PASSIVE products of C3S SM v202212 were also compared to the reference datasets described in Section 2. As expected, the COMBINED product overall outperforms the PASSIVE and ACTIVE products as shown in Figure 44 for the comparison to ERA5-Land.

Figure 44: Intercomparison of the COMBINED, ACTIVE, PASSIVE product of C3S v202212, with ERA5-Land as the reference – plots shown are for Pearson's R (top) and ubRMSD (bottom). Only common locations and timestamps between all inter-compared products are considered.

Figure 45 and Figure 46 show differences in correlation and ubRMSD with respect to ERA5-LAND respectively, between the COMBINED and ACTIVE or PASSIVE products. The COMBINED product usually outperforms the ACTIVE and PASSIVE products. The difference is especially visible in deserts, where the ACTIVE data perform poorly due to the negative impact of subsurface scattering on soil moisture retrieval. The COMBINED product falls back to observations from passive sensors in this case. The opposite is visible around densely vegetated areas, where passive sensors are more negatively affected than active ones, and the COMBINED product can fall back to radar-based soil moisture retrievals.

Figure 45: Difference in correlation with ERA5-LAND between the COMBINED and ACTIVE (top), resp. COMBINED and PASSIVE (bottom) products of C3S SM v202212.

Figure 46: Difference in ubRMSD with ERA5-LAND between the COMBINED and ACTIVE (top), resp. COMBINED and PASSIVE (bottom) products of C3S SM v202212.

A.2. Differences in long-term SM trends between the three products

Long-term trends in SM data sets are key to understanding the impact of climate change on global soil moisture. With more than 40 years of data, the assessment of long-term trends in C3S SM is possible. The COMBINED product has been used for analyses like this before (Dorigo et al. 2012). Ideally, one would expect similar trends in the ACTIVE, COMBINED and PASSIVE products. However, as Figure 47 shows there are differences and trends are often contradictory. Trends in ASCAT are known to be affected by long-term changes in land cover that the retrieval algorithm does not sufficiently account for, and potentially affect the COMBINED product as well. A trend correction algorithm is currently being developed at EUMETSAT H-SAF.

Figure 47: Differences in long-term (Theil-Sen median) trends (1991-2020). A Mann-Kendall significance test was applied and only statistically significant trends are shown here. Note: different scales are shown on these maps such that the colors shown in each case are similar to one another; this is a result of the different absolute values provided in each data product.

The current suggestion is to use the COMBINED product for long-term studies. However, different trends indicate insufficient inter-calibration of some sensors or artificial trends within single products, as is demonstrated in Section A.3.

A.3. Wetting trends in the ASCAT data

An artificial wetting trend is found in ASCAT SSM. This affects the ACTIVE product (and the COMBINED product to a lesser extent) of C3S SM, as well as derived anomalies. Figure 48 shows anomalies over Europe for the year 2019 as in the ESA CCI SM v4 dataset. Note that this issue is still affecting ASCAT SSM and therefore C3S SM ACTIVE and to a lesser extent COMBINED at version v202212. Hence the issues described here still apply to the latest version. The right column shows the same anomalies when using an experimental version of ASCAT SSM with trend correction in the backscatter dry and wet reference in ESA CCI SM. Artificial trends in the radar observed backscatter signal are probably due to landcover changes and/or RFI (Ticconi et al. 2017). Cities, in particular, stand out in the SM anomaly maps of the uncorrected ACTIVE product. The issue was first detected over Europe, but also affects other regions worldwide.

It is also noted that the issue seems most prominent in spring, and therefore may be (partly) related to the vegetation correction used in the ASCAT product.

The trend-corrected product of ASCAT SSM will be used in ESA CCI SM and C3S SM, if officially released and provided by H-SAF.

Figure 48: Change in ESA CCI SM v4 annual anomalies (for the year 2019) due to backscatter trend correction in ASCAT SSM. The climatological period for all plots is from 1991-2010. Note the difference scales for each of the three products.

Annex B: Detailed comparison of C3S v202212 against C3S v202012

B.1. Introduction

This Annex provides a detailed comparison of the newest dataset version (v202212) against the previous version (v202012). The aim is to determine that the dataset has been made to specification. Differences between the products, which may be of interest to users when using the data in their applications, are highlighted.

B.2. Comparison of data coverage

Significant changes in how data is masked were introduced in the new version. The main reasons for changes in coverage are:

• Four additional sensors and daytime observations were added to fill gaps in the product. This is a significant increase in terms of available data, allowing the filling of many gaps, except those due to frozen soils (see next point).
• A cross-flagging and -masking approach was introduced to improve consistency in coverage between the satellite products before merging them. This means that observations from all sensors are discarded because of potentially frozen soils as soon as at least one satellite classifies the soil as frozen. A downside of this is, that insufficient surface temperature calibration of one sensor can therefore affect the coverage of the whole product. The upside is a stricter masking of frozen soils that otherwise could degrade the product performance significantly.

The number of valid observations available in each product version has been compared for the ACTIVE, PASSIVE and COMBINED products. Figure 49 and Figure 50 show changes in the number of valid observations for the three daily products (by latitude and month). It can be seen that the coverage over time in PASSIVE and COMBINED data increased significantly compared to the previous version. This is due to the inclusion of four additional passive satellites as well as the use of day-time observations for all passive satellites to fill gaps in the night-time retrievals. The only relevant decrease in coverage is found in the period between 2015 to 2020. This is mainly due to over-flagging of frozen soils in the newly added FengYun-3C (hence the seasonal pattern) as well as cross-flagging between sensors. However, this data loss is comparably small and does mainly affect areas in the transition zone from frozen to unfrozen ground.

Figure 49: Change in (relative) number of valid observations between v202012 and v202212 for the COMBINED product. Green indicates an increase compared to the previous version, red a decrease in observations.

The ACTIVE product shows an overall decrease in coverage compared to the previous version. This is due to the newly introduced cross-flagging between active and passive sensors. ASCAT observations were flagged less rigorously than LPRM observations in the past. The LPRM-based detection of frozen soils is now also applied to ASCAT in the new version, leading to significantly more masked observations, especially in the northern hemisphere in ACTIVE.

ASCAT-C was introduced in v202212 (available after April 2019). ASCAT-A and –B already provided good overall global coverage for a daily product in previous versions. The additional sensor, therefore, has led to only a small increase in coverage after 2021, when ASCAT-A operation was discontinued. To ensure uninterrupted data production, it was however crucial to include data from at least one additional active sensor for the event of a failure of one satellite. This is achieved with the inclusion of ASCAT-C.

Figure 50: Change in (relative) number of valid observations between v202012 and v202212 for the ACTIVE (top) and PASSIVE (bottom) products. Green indicates an increase compared to the previous version, red a decrease in observations.

The same differences are shown spatially in Figure 51. This shows the exact locations where data points were gained or lost (due to masking) between the versions. While there was an overall increase in data coverage globally, in some locations fewer data points are found in v202212 than in v202012. This is due to the stricter cross-flagging applied in v202212, and the potential (over)flagging of frozen soils found for FY-3C. Overall in COMBINED and PASSIVE, a significant increase in observations was achieved. Decreases are found in mountainous areas, bare soils and high latitudes mainly, where a stricter flagging of potentially erroneous measurements is generally preferred.

The decrease in the total number of data points in ACTIVE is most obvious. However, this is expected and due to the cross-flagging between active and passive sensors This is in line with the data coverage of ESA CCI SM v7. An increase is found in ACTIVE in areas that are not affected by cross-flagging, corresponding to the temporal extension of the product.

Figure 51: Change in number of observations (daily) in C3S SM v202212 compared to v202012 in the period after 1991-01-01 for the COMBINED (top), ACTIVE (middle) and PASSIVE (bottom) products

B.3. Comparison of time series

The locations, which are compared in Figure 52 for the different product versions, are the same as in Figure 37. Overall, the products appear to be similar at these locations. For most points the expected increase in data coverage is found; also the temporal extension of the product is visible. GPI 733335 shows some reduced outlier values at the beginning of 1983 and 2018 in the new version.

Figure 52: Time series of soil moisture (in [m³/m³]) for the different land cover classes considered (GPI locations shown in Figure 37). Showing the data for the COMBINED product from v202212 and v202012 aggregated to 10-day time steps.

B.4. Comparison of daily images

Daily images for 2019-07-01 for each of the ACTIVE, PASSIVE and COMBINED products have been compared for C3S v202212 and v202012 (difference between them). Figure 53 shows that large differences are found in the PASSIVE product (due to the new sensors and LPRM version used), some smaller changes are found for COMBINED for the same reason.

Figure 53: Absolute difference in soil moisture between C3S SM v202212 and v202012 for the daily COMBINED (top), ACTIVE (middle) and PASSIVE (bottom) product on 2019-07-01.

B.5. Comparison of global statistics

To demonstrate the differences between the previous C3S version (v202012) and the current dataset (v202212), global statistics have been computed for each dataset version and are provided in Table 5. These are for the period after 2015-04-01 and are based on the provided monthly mean values.

It can be seen that COMBINED remained similar in terms of value range and mean/median. This is expected as the same scaling reference is used as in previous versions. The same applies for ACTIVE. The largest differences are found for PASSIVE, where due to the new LPRM version soil moisture (especially in sub-arctic latitudes) overall decreased. The standard deviation however increased, which is also due to the addition of more sensors as well as day-time observations in the product.

Table 5: Dataset statistics for the different C3S versions CDRs for the latest merging period for each product. The numbers given are the mean values across all points (first temporally, then spatially).

Annex C – Validation of Monthly and Dekadal Data

In addition to the daily C3S SM files, monthly and 10-day (dekadal) averaged files are provided. These include the number of averaged observations per file, the average SM (same unit as the daily data) and fields for the sensors and frequency bands that are merged.

Evaluation of these data was carried out with ERA5-Land as the reference (monthly and 10-day mean values from the original hourly data between 1981 and 2022). The absolute values were used in all cases. A mean/standard deviation scaling is applied to all satellite products in this validation to make them comparable.

Correlations between monthly values (Figure 54) are higher than between daily ones. The COMBINED product also outperforms the ACTIVE and PASSIVE in this comparison, reaching a median R of 0.67 (median ubRMSD of 0.0375 m³/m³ and therefore below the 0.04 m³/m³ KPI target). Also on the monthly time scale, the ACTIVE product shows low correlations with ERA5-Land in deserts, where subsurface scattering affects the soil moisture retrieval in the satellite data.

The same overall results are found for the dekadal data shown in Figure 55.

Figure 54: Correlation (top) and ubRMSD (bottom) between monthly aggregated C3S SM products and ERA5-Land.

Figure 55: Correlation (top) and ubRMSD (bottom) between 10-day (dekadal) aggregated C3S SM products and ERA5-Land.

Notably, there is currently no threshold for a minimum number of observations per location in the daily images that are averaged. The representativeness of averaged images for the actual 10-day and monthly SM conditions depends on the data coverage for each aggregation period. This should be considered - especially for early periods of C3S SM and for areas with low data density - when using the aggregated files.

References

(JCGM), J. C. F. G. I. M. 2008. International vocabulary of metrology — Basic and general concepts and associated terms (VIM). VIM3: International Vocabulary of Metrology, 3, 104.

Dorigo, W., de Jeu, R., Chung, D., Parinussa, R., Liu, Y., Wagner, W., & Fernández-Prieto, D. (2012). Evaluating global trends (1988–2010) in harmonized multi-satellite surface soil moisture, 39

Dorigo, W., Himmelbauer, I., Aberer, D., Schremmer, L., Petrakovic, I., Zappa, L., Preimesberger, W., Xaver, A., Annor, F., Ardö, J., Baldocchi, D., Bitelli, M., Blöschl, G., Bogena, H., Brocca, L., Calvet, J.C., Camarero, J.J., Capello, G., Choi, M., Cosh, M.C., van de Giesen, N., Hajdu, I., Ikonen, J., Jensen, K.H., Kanniah, K.D., de Kat, I., Kirchengast, G., Kumar Rai, P., Kyrouac, J., Larson, K., Liu, S., Loew, A., Moghaddam, M., Martínez Fernández, J., Mattar Bader, C., Morbidelli, R., Musial, J.P., Osenga, E., Palecki, M.A., Pellarin, T., Petropoulos, G.P., Pfeil, I., Powers, J., Robock, A., Rüdiger, C., Rummel, U., Strobel, M., Su, Z., Sullivan, R., Tagesson, T., Varlagin, A., Vreugdenhil, M., Walker, J., Wen, J., Wenger, F., Wigneron, J.P., Woods, M., Yang, K., Zeng, Y., Zhang, X., Zreda, M., Dietrich, S., Gruber, A., van Oevelen, P., Wagner, W., Scipal, K., Drusch, M., & Sabia, R. (2021). The International Soil Moisture Network: serving Earth system science for over a decade. Hydrol. Earth Syst. Sci., 25, 5749-5804

Dorigo, W., Van Oevelen, P., Wagner, W., Drusch, M., Mecklenburg, S., Robock, A., & Jackson, T. (2011). A new international network for in situ soil moisture data. Eos, 92, 141-142

Dorigo, W., Wagner, W., Albergel, C., Albrecht, F., Balsamo, G., Brocca, L., Chung, D., Ertl, M., Forkel, M., Gruber, A., Haas, E., Hamer, P.D., Hirschi, M., Ikonen, J., de Jeu, R., Kidd, R., Lahoz, W., Liu, Y.Y., Miralles, D., Mistelbauer, T., Nicolai-Shaw, N., Parinussa, R., Pratola, C., Reimer, C., van der Schalie, R., Seneviratne, S.I., Smolander, T., & Lecomte, P. (2017). ESA CCI Soil Moisture for improved Earth system understanding: State-of-the art and future directions. Remote Sensing of Environment

Dorigo, W.A., Xaver, A., Vreugdenhil, M., Gruber, A., Hegyiová, A., Sanchis-Dufau, A.D., Zamojski, D., Cordes, C., Wagner, W., & Drusch, M. (2013). Global Automated Quality Control of In Situ Soil Moisture Data from the International Soil Moisture Network. Vadose Zone Journal, 12, 0

ENTEKHABI, D., REICHLE, R. H., KOSTER, R. D. & CROW, W. T. 2010. Performance Metrics for Soil Moisture Retrievals and Application Requirements. Journal of Hydrometeorology, 11, 832-840.

Gruber, A., Dorigo, W.A., Crow, W., Wagner, W., & Member, S. (2017). Triple Collocation-Based Merging of Satellite Soil Moisture Retrievals, 1-13

Gruber, A., Dorigo, W.A., Zwieback, S., Xaver, A., & Wagn, W. (2013). Characterizing Coarse-Scale Representativeness of in situ Soil Moisture Measurements from the International Soil Moisture Network. Vadose Zone Journal, 12

Gruber, A., De Lannoy, G., Albergel, C., Al-Yaari, A., Brocca, L., Calvet, J. C., Colliander, A., Cosh, M., Crow, W., Dorigo, W., Draper, C., Hirschi, M., Kerr, Y., Konings, A., Lahoz, W., Mccoll, K., Montzka, C., Muñoz-Sabater, J., Peng, J., Reichle, R., Richaume, P., Rüdiger, C., Scanlon, T., Van Der Schalie, R., Wingeron, J. P. & Wagner, W. (2020). Validation practices for satellite soil moisture retrievals: What are (the) errors? Remote Sensing of Environment, 244, 111806.

Hersbach, H., Bell, B., Berrisford, P., Hirahara, S., Horányi, A., Muñoz-Sabater, J., Nicolas, J., Peubey, C., Radu, R., Schepers, D., Simmons, A., Soci, C., Abdalla, S., Abellan, X., Balsamo, G., Bechtold, P., Biavati, G., Bidlot, J., Bonavita, M., De Chiara, G., Dahlgren, P., Dee, D., Diamantakis, M., Dragani, R., Flemming, J., Forbes, R., Fuentes, M., Geer, A., Haimberger, L., Healy, S., Hogan, R.J., Hólm, E., Janisková, M., Keeley, S., Laloyaux, P., Lopez, P., Lupu, C., Radnoti, G., de Rosnay, P., Rozum, I., Vamborg, F., Villaume, S., & Thépaut, J.-N. (2020). The ERA5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146, 1999-2049

Liu, Y.Y., Dorigo, W.A., Parinussa, R.M., De Jeu, R.A.M., Wagner, W., McCabe, M.F., Evans, J.P., & Van Dijk, A.I.J.M. (2012). Trend-preserving blending of passive and active microwave soil moisture retrievals. Remote Sensing of Environment, 123, 280-297

Muñoz-Sabater, J., Dutra, E., Agustí-Panareda, A., Albergel, C., Arduini, G., Balsamo, G., Boussetta, S., Choulga, M., Harrigan, S., Hersbach, H., Martens, B., Miralles, D.G., Piles, M., Rodríguez-Fernández, N.J., Zsoter, E., Buontempo, C., & Thépaut, J.N. (2021). ERA5-Land: a state-of-the-art global reanalysis dataset for land applications. Earth Syst. Sci. Data, 13, 4349-4383

Pasik, A., Gruber, A., Preimesberger, W., De Santis, D., & Dorigo, W. (2023). Uncertainty estimation for a new exponential filter-based long-term root-zone soil moisture dataset from C3S surface observations. EGUsphere, 2023, 1-32

PEEL, M. C., FINLAYSON, B. L. & MCMAHON, T. A. 2007. Updated world map of the Köppen-Geiger climate classification. Hydrol. Earth Syst. Sci., 11, 1633-1644.

Plummer, S., Lecomte, P., & Doherty, M. (2017). The ESA Climate Change Initiative (CCI): A European contribution to the generation of the Global Climate Observing System. Remote Sensing of Environment

Ticconi, F., Anderson, C., Figa-Saldana, J., Wilson, J.J.W., & Bauch, H. (2017). Analysis of Radio Frequency Interference in Metop ASCAT Backscatter Measurements. Ieee Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 10, 2360-2371

Wagner, W., Dorigo, W., de Jeu, R., Fernandez, D., Benveniste, J., Haas, E., & Ertl, M. (2012). Fusion of Active and Passive Microwave Observations To Create an Essential Climate Variable Data Record on Soil Moisture. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, I-7, 315--321

Related articles

• Page:
  SM 2020: Target Requirements and Gap Analysis Document (TRGAD)
• Page:
  SM v202012: Algorithm Theoretical Basis Document (ATBD)
• Page:
  SM v201912: Product Quality Assessment Report (PQAR)
• Page:
  SM v 201912: Algorithm Theoretical Basis Document (ATBD)
• Page:
  SM v201812: Algorithm Theoretical Basis Document (ATBD)