Last modified on Mar 18, 2024 14:51

Contributors: W. Dorigo (WD) (TU Wien), T. Scanlon (TS) (TU Wien), W. Preimesberger (WP) (TU Wien), R. Kidd (RK) (EODC)

Issued by: EODC/R. Kidd

Date: 11/07/2019

Ref: C3S_312b_Lot4.D2.SM.1_20190509_PQAR_v1.3

Official reference number service contract: 2018/C3S_312b_Lot4_EODC/SC1

History of modifications

Related documents

Acronyms

General definitions

Accuracy: The closeness of agreement between a measured quantity value and a true quantity value of a measure and ((JCGM), 2008). The metrics used here to represent accuracy are correlation and unbiased Root Mean Square Difference (ubRMSD). These metrics are commonly used throughout the scientific community as measures of accuracy (Entekhabi et al., 2010)

Bias: Estimate of a systematic measurement error ((JCGM), 2008).

Error: Measured quantity value minus a reference quantity value ((JCGM), 2008).

Precision: Closeness of agreement between indications or measured quantity values obtained by replicate measurements on the same or similar objects under specified conditions ((JCGM), 2008).

Quality Assurance: Part of quality management focused on providing confidence that quality requirements will be fulfilled (BSI, 2015).

Stability: Property of a measuring instrument whereby its metrological properties remain constant in time ((JCGM), 2008). Note that for earth observation activities, "measuring instrument" can be translated as "retrieved variable" and "metrological properties" can be interpreted as "variability of retrieved variable". Alternatively, the Global Climate Observing System (GCOS) (WMO, 2016) defines stability as the extent to which the uncertainty of measurement remains constant in time; the GCOS requirements are stated as the maximum acceptable change in systematic error (usually per decade).

Scope of the document

The purpose of this document is to describe the results of the Quality Assurance (QA) for the soil moisture product developed by TU Wien, EODC, VanderSat and AWST for the Copernicus Climate Change (C3S) service. The product version assessed in this report is v201812.0.0, which was produced in January 2019.

The C3S soil moisture product suite provides PASSIVE, ACTIVE and COMBINED (passive + active) microwave soil moisture products on a daily, dekadal (10-days) and monthly basis. The data is provided in a regular 0.25 degree grid based on the World Geodetic System 1984 (WGS 84) reference system. The product is available globally between November 1978 and present day (for PASSIVE and COMBINED) and 1991 and present day (for ACTIVE). For details about the products, we refer to the Product User Guide and Specification (PUGS) (Dorigo et al., 2019a).

This document presents the results of QA activities that have been undertaken for the current Thematic Climate Data Record (TCDR) dataset (v201812.0.0). The Interim Climate Data Record (ICDR) datasets are not currently assessed. However, note that, to achieve maximum consistency between TCDR and ICDR, both products use the same Level 2 products (based on Near Real Time (NRT) data streams) and merging algorithms and thus have very similar quality characteristics.

A brief summary of the methodology used (described fully in the Product Quality Assurance Document (PQAD) (Dorigo et al., 2019d)) is provided. The results described here are primarily for the COMBINED daily product, however, an assessment of the ACTIVE and PASSIVE products were undertaken and the main findings of these assessments are presented in the report.

Executive summary

The purpose of the Product Quality Assessment Report (PQAR) is to describe the product QA results for the soil moisture product developed by TU Wien, EODC, VanderSat and AWST for the C3S service. The production of the product has been funded by C3S, a service which is managed by the European Centre for Medium-Range Weather Forecasting (ECMWF) on behalf of the European Commission (EC). The product version assessed in this report is v201812.0.0, which was produced in January 2019.

The document presents the results of the quality assessments undertaken for the product including the accuracy and stability assessment of the product. This document is applicable to the QA activities performed on the version of the TCDR v201812.0.0. Currently, the assessment does not cover ICDRs but due to the high consistency between TCDR and ICDR (both products use the same Level 2 products, based on NRT data streams, and merging algorithms), the QA assessment of the last years of the TCDR can be readily transferred to the ICDR.

The QA results broadly include the following parts: accuracy assessment, stability assessment, demonstration of uncertainty estimates, comparison to previous versions of the product, and a completeness / consistency assessment. The first two sections focus on demonstrating that the Key Performance Indicators (KPIs) set for the product are met. Note these KPIs take into account GCOS and user requirements for the product. In addition, in this version of the document, there is also a detailed assessment of the product against the previous version as well as details of the changes in the input data streams and how this may affect the product.

Accuracy Assessment: In general, there is a slight variability in the correlation between the datasets, with correlations ranging from between 0.4 to above 0.8 depending on the conditions and the locations of the in-situ stations used. The ubRMSD, which can be directly taken as a measure of accuracy, is demonstrated to be below 0.10 m³ / m³ for all of the different conditions analysed. Therefore, the KPIs for accuracy have been met for in-situ observations. The global comparison against the Global Land Data Assimilation System (GLDAS) Noah v2.1 has shown expected results, with the KPI threshold of 0.1 m³ / m³ met in most regions (the exception being areas with high topographic complexity). However, for the ECMWF Reanalysis (ERA) 5 comparison there are several unexpected spatial patterns and, in many areas, the KPI threshold target is not met and further investigation is required.

Stability Assessment: The stability of the C3S product has been assessed in terms of the change in accuracy (when compared to the GLDAS v2.1 product in the period 2000-01-01 to 2018-12-31). The accuracy between the products (ubRMSD) has been calculated per year and the trends in the accuracy calculated. Spatially, the assessment shows that there are some significant changes in the accuracy in the northern boreal areas, with the most stable regions being arid areas. The KPI threshold for stability of 0.05 m³ / m³ / y is met when assessed using this method for most geographical regions.

A comparison to previous products has been provided, comparing the C3S products to previous, similar European Space Agency (ESA) Climate Change Initiative (CCI) products. The assessment demonstrates that the correlation between the in-situ and satellite-derived products has improved between the product versions.

The spatial and temporal coverage of the product has been presented in terms of the number of valid (unflagged) observations available. It is shown that the coverage is better in Europe, South Africa and the continental United States (US) than in some other parts of the world. In addition, the improvement in the availability of data post-2007 (as new sensors became available) is shown.

Further, detailed assessment of the products have been undertaken, in particular for the ACTIVE and PASSIVE products as well as a detailed comparison against the previous product version. These assessments revealed that there are the following potential issues with the dataset:

• PASSIVE missing data: The PASSIVE product is missing data from the Advanced Microwave Scanning Radiometer 2 (AMSR2) in the latest merging period. This needs to be rectified by reprocessing the dataset v201812 prior to inclusion in the Climate Data Store (CDS).
• PASSIVE timeseries drop: After 2011, there seems to be a significant drop in soil moisture values for several land cover types. The reason for this has not been determined, however, it may be due to the scaling used in generation of the PASSIVE dataset. This will be investigated during the development of the new product (within the CCI+ programme).
• ACTIVE wetting trends: There appears to be unrealistic wetting trends in the ACTIVE product which are also impacting the COMBINED product (due to the inclusion of the Advanced Scatterometer (ASCAT) data). These trends need to be investigated. This will be investigated during the development of the new product (within the CCI+ programme).
• ERA5 assessment: The assessment against the ERA5 dataset has shown unexpected results and provides a ubRMSD much higher than the threshold KPIs. The comparison method, in terms of masking of highly vegetated areas and frozen soils / snow covered conditions needs to be reconsidered. The new method will be implemented at the next version of this document (for v201912 of the product).

1. Product validation methodology

1.1. Validated products

C3S soil moisture provides passive (named PASSIVE), active (ACTIVE) and passive + active (COMBINED) soil moisture products on a daily, dekadal (10-days) and monthly basis. The time periods over which each sensor is used are provided in Table 1. The data is provided in a regular 0.25 degree grid based on the WGS 84 reference system. The product is available globally between November 1978 and present day (for PASSIVE and COMBINED) and 1991 and present day (for ACTIVE). The product has been produced by TU Wien, Vandersat, EODC and AWST.

Table 1: Blending periods for the soil moisture product (ACTIVE, PASSIVE and COMBINED)

*The [SSM/I, TMI, SSM/I] period is latitudinally divided into [90S, 37S] and [90N, 37N] for SSM/I, and the region in between for TMI.

The C3S soil moisture product is generated from a set of passive microwave radiometers (Scanning Multichannel Microwave Radiometer (SMMR), Special Sensor Microwave Imager (SSM/I), TRMM Microwave Imager (TMI), Advanced Microwave Scanning Radiometer-Earth Observing System (AMSR-E), WindSat Spaceborne Polarimetric Microwave Radiometer (WindSat), AMSR2 and Soil Moisture and Ocean Salinity (SMOS)) and active microwave scatterometers (Active Microwave Instrument - Windscat (AMI-WS) and ASCAT (Metop-A and Metop-B)). The “ACTIVE product” and the “PASSIVE product” are created by fusing scatterometer and radiometer Level 2 soil moisture products, respectively; the “COMBINED product” is created by fusing Level 2 soil moisture products from both sensor types. Data files are provided as NetCDF-4 classic format as daily, dekadal and monthly images and comply with CF-1.6 conventions¹.

A detailed description of the product generation of C3S v201812.0.0 is provided in the Algorithm Theoretical Basis Document (ATBD) (de Jeu et al., 2019b) with further information on the product given in the PUGS (Dorigo et al., 2019a). The underlying algorithm is based on that used in the generation of the publically released ESA CCI version 4.4 product which is described in relevant documents ((Plummer et al., 2017), (Wagner et al., 2012), (Liu et al., 2012), (Dorigo et al., 2017a)).  In addition, detailed provenance traceability information can be found in the metadata of the product.

The C3S soil moisture product comprises a long-term data record called a Thematic Climate Data Record (TCDR) which runs from 1978 PASIVE and COMBINED) or 1991 (ACTIVE) to December 2018.  This TCDR is updated every dekad (approximately every 10 days) in an appended dataset called an Interim Climate Data Record (ICDR).  The theoretical algorithm and the processing implemented in the TCDRs and ICDRs are exactly the same and the data provided is consistent between them.

This current document is applicable to the QA activities performed on the version of the TCDR v201812.0.0 (produced in January 2019). The COMBINED daily product is the focus of the validation activities presented in this report, however, the ACTIVE and PASSIVE daily products have also been validated and a summary of the outcomes is provided in section 5.

1.2. Description of validating datasets

A combination of in-situ and global reference datasets are utilised to assess the quality of the C3S soil moisture product. A list of the datasets utilised is provided in Table 2 with further details provided in the PQAD (Dorigo et al., 2017b).

Table 2: Datasets utilised in the assessment of the data product

1.3. Description of product validation methodology

1.3.1. Method Overview

The methodology used in the assessment of the soil moisture product is described in (Dorigo et al., 2017b). The methodology, including details of the validated products and the validating datasets are described briefly here in the relevant sections; conversely, a summary of the validation results may be found in (Dorigo et al., 2017b) for the previous dataset version (v201706).

The quality assessment includes the following:

1. Assessment against in-situ observations from the ISMN (Section 2.1),
2. Assessment against Land Surface Models (LSMs) including GLDAS and ERA-5 (Section 2.2),
3. Stability analysis through monitoring of accuracy trends and dataset statistics (Section 2.3),
4. Stability analysis through breakpoint detection (Section 2.4),
5. Assessment of the spatial and temporal completeness and consistency of the product (Section 2.5),
6. Analysis of timeseries at selected locations (Section 2.6), and,
7. Analysis of the uncertainty information provided with the dataset (Section 2.7).

Additional information is also provided in the following sections of the report:

1. Section 5: The main body of the report considers mainly the COMBINED TCDR, therefore, a separate Annex provides information on the validation results of the ACTIVE and PASSIVE products; summarising the key findings from these activities.
2. Section 6: The main body of the report provides some comparison between the current C3S version (v201812) and the previous version (v201706). This section provides a more complete analysis to demonstrate that the data has been made correctly and show any differences between the products.
3. Section 7: There have been changes in the data streams between the current product version (v201812) and the previous version (v201706). This section analyses the impact of these changes.

1.3.2. Quality Assurance for Soil Moisture

The analysis presented in this report has been undertaken using software specifically designed for the C3S soil moisture project. However, in parallel, an online validation service (Quality Assurance for Soil Moisture (QA4SM)) has been developed which undertakes similar tasks.

The QA4SM service is an online validation service which allows the traceable validation of satellite derived soil moisture products. Currently supported datasets include C3S (v201706, v201812), SMAP v5, the H-SAF ASCAT product H115, the ESA CCI SM product (v04.4) and the SMOS IC product. The comparisons can be carried out against both ISMN and GLDAS (note that ERA5 will be added shortly). The service provides different options for the filtering of datasets and for scaling the datasets to each other.

The QA4SM service is continuously under development and does not currently have all of the features required to undertake a full validation of the C3S data as required for the validation activities presented in the current document. Therefore, here we show the results which are available to demonstrate what can be achieved with the system. As the QA4SM service develops, the analysis presented in this report will be updated with results from this system at the generation of new product versions.

1.3.3. Planned enhancements to the method

The quality assurance methods for soil moisture are continuously under development. This section briefly details enhancements to the methods envisaged for the future and will be implemented at the generation of new product versions.

Committed area mask
In future versions of this assessment, it is proposed that a committed area mask, akin to that used in the validation of the H-SAF soil moisture project (see Figure 1), will be used. This mask is used to better understand the performance of the product globally. This mask is based on several criteria and thresholds, each of which will be considered in the generation of a C3S specific committed.

Figure 1: Committed area used by the H-SAF soil moisture project. The mask is based on a combination of vegetation optical depth thresholds, snow / ice masking and land cover types.

Consistent masking of the validation datasets
Currently, the masking of the datasets used in the validation is undertaken dependent on the flags available within the separate data products. For example, the GLDAS masking is done based on no snow cover flags. However, between the different validations, there is no consistency on which locations / timestamps are masked. The introduction of consistent masking would help improve the inter-comparison of the validation results using data from different sources.

Assessment of the discontinuities in ERA5
It is known that there are discontinuities in the ERA5 soil moisture variables due to the splitting of the processing into chunks. The assessment presented here shows that these discontinuities appear to not affect the upper most soil layer and therefore, the assessment of the C3S product against this data is appropriate. However, this assertion has been made solely on the visual interpretation of the ERA 5 timeseries and further analysis should be carried out to determine if this is the case.

2. Validation results

2.1. Accuracy – Comparison against ISMN

2.1.1. Introduction

The C3S dataset has been compared to the ISMN dataset and the correlations and ubRMSD between the datasets calculated.

In addition to an overall comparison processed using the QA4SM service (Section 2.1.3), the comparison is also undertaken for different attributes of the soil moisture data (provided as metadata within the ISMN dataset). These are: sensor depth (Section 2.1.4), soil texture (Section 2.1.5), Köppen-Geiger climate classes (Section 2.1.6) and land cover (Section 2.1.7). For further information on the origin of these attributes, see Dorigo et al. (2011).

The aim of assessing the accuracy for different attributes is to demonstrate the performance of the product under different conditions and demonstrate that KPIs DX.1 (where X is equal to 1 – 6) are met (see the PQAD (Dorigo et al., 2019d), Table 2, for the list of KPIs; repeated in this document at Table 6).

2.1.2. ISMN data and comparison settings

The ISMN data used in the assessments presented in this document were downloaded from the ISMN data portal⁵ on 2019-02-22 (hence this is referred to as v20190222). The full list of the networks used in the assessment can be found in the PQAD (Dorigo et al., 2019d). The dataset consists of 751 stations over 29 networks, which are matched to a total of 355 grid point indexes (gpis) in the C3S grid.

Where there is more than one station available within a grid cell, a simple average of the station data is taken prior to comparison to the C3S timeseries for all assessments except the overall comparison. In the overall comparison (processed through the QA4SM service) no averaging of the ISMN timeseries is undertaken. Instead, a comparison is undertaken for each ISMN station against each nearest neighbour C3S gpi, hence, in these results, some gpis are over-represented.

To calculate the metrics for each assessment, the settings summarised in Table 3 are used. In summary, the datasets are spatially and temporally matched, filtered for high quality observations and the C3S data is scaled to the ISMN data using mean – standard deviation scaling. The metrics are then calculated using observations in the period 1978-11-01 to 2018-12-31.

Table 3: Settings used in the assessment of the C3S soil moisture product against the ISMN

2.1.3. Overall Comparison

The comparison of C3S v201812 has been processed using the QA4SM service against ISMN v20190222. The global map (Figure 2) shows the ubRMSD for each ISMN station to the nearest C3S grid cell; correlation (Pearson's) is shown in Figure 3. These figures show the expected spatial patterns, with high correlations and low ubRMSD seen at most ISMN locations.

Figure 2: ubRMSD between C3S v201812 and ISMN v20190222 for soil depths of 0 – 5 cm. Generated using the QA4SM service.

Figure 3: Correlation (Pearson's) between C3S v201812 and ISMN v20190222 for soil depths of 0 – 5 cm. Generated using the QA4SM service.

2.1.4. Soil depth

The correlation between the C3S dataset and the in-situ datasets are presented in Figure 4 for two different surface soil moisture depths (0 to 5 cm and 5 to 10 cm).

The deeper sensors have a lower correlation and higher ubRMSD than the sensors at shallower depths. This is as expected as the product represents the first few centimetres of the soil surface (approx.). However, it may also be attributed to the decreasing errors associated with the ISMN data at these deeper depths (noting that the relation seems to break at depths > 1.00 m) (Gruber et al., 2013).

Figure 4: Correlation (left) and ubRMSD (right) between C3S v201812 and ISMN for soil depths of 0 – 5 cm and 5 – 10 cm. The boxplots show the mean value and interquartile range.

2.1.5. Soil texture

The correlation and ubRMSD between the C3S dataset and the in-situ datasets are presented in Figure 6 for the different soil textures (fine, medium and coarse) (stratification provided from the ISMN dataset (Dorigo et al., 2011) and shown in Figure 5).

The product appears to perform best for medium texture soils in terms of correlation but interestingly this is the worst category for the ubRMSD. The coarse texture soils have relatively few observations available, therefore the results for this soil texture are not considered reliable.

Figure 5: Soil texture classification used for the ISMN comparison. The percentage clay, silt and sand are taken from the ISMN metadata which in turn is retrieved from the HWSD.

Figure 6: Correlation (Pearson) and ubRMSD between C3S v201812 and ISMN for different soil texture classes. The boxplots show the mean value and interquartile range.

2.1.6. Köppen-Geiger climate classes

The correlation and ubRMSD between the C3S dataset and the in-situ datasets are presented in Figure 7 for different Köppen-Geiger classes (BSx⁷, Csx / Dsx⁸ and Cfx / Dfx⁹), a global map of which is shown in Figure 8 (stratification provided from the ISMN metadata (Dorigo et al., 2011)).

The correlation between the in-situ measurements and the C3S product is similar across all Köppen-Geiger classes, with Csx / Dsx performing the best in terms of correlation and second best in terms of ubRMSD. The ubRMSD can be taken as a measure of accuracy; the graphs show that in this case, the mean value for all of the different climate classes are under the 0.1 m³ / m³ threshold set out in the KPIs; therefore the KPIs are achieved.

Figure 7: Correlation and ubRMSD between C3S v201812 and ISMN for different climate classes. The boxplots show the mean value and interquartile range.

Figure 8: Köppen-Geiger classes. The classes used in this assessment are BSx, Csx / Dsx and Cfx / Dfx. The Figure is taken from http://hanschen.org/koppen.

2.1.7. Landcover classes

The correlation between the C3S dataset and the in-situ datasets are presented in Figure 9 for two different land cover classes (grassland vs. tree-cover) (stratification provided from the ISMN dataset (Dorigo et al., 2011)).

The ubRMSD can be taken as a measure of accuracy and the KPIs specially state the acceptable accuracy level of the product is between 0.01 and 0.1 m³ / m³ for different land cover types. The worse-case ubRMSD shown here is under 0.1 m³ / m³ (median) for tree cover although it is noted that the IQR is over to the threshold of 0.1 m³ / m³. However, the median remains below the threshold and therefore, the KPIs are met.

Figure 9: Correlation and ubRMSD between C3S v201812 and ISMN for different land cover classes. The boxplots show the mean value and interquartile range.

2.1.8. Seasonal variation assessment

The results presented in this section are related to ESA CCI SM v04.2. The C3S soil moisture TCDR dated December 2018 (v201812) is the same (scientifically, algorithmically and programmatically) as the ESA CCI SM v04.5 product; the only difference between them is the use of different ASCAT data streams (to ensure consistency with the NRT product generation).

The C3S dataset has been compared to ISMN and the assessment split by seasons. The in-situ data chosen is constrained to the USA. The data is not masked for common time-steps and not scaled to the in-situ data. In addition, constraints in data selection were applied to ensure that at least 10 % of the time series is not NA, the p-value is < 0.05 and the calculated correlation is positive.

The correlation between the in-situ dataset and the C3S dataset has been determined for the average of the Scaled Absolute Values (ABS) over consecutive four-year periods between 1997 and 2012, similar to that undertaken in (Dorigo and Gruber, 2015), (Albergel et al., 2013). The results for the entire year and three-monthly periods are presented in Figure 10 with maps of the same data presented in Figure 11.

The correlations and their spread between the in-situ measurements and the C3S product are similar across the seasons, showing a slight improvement in the autumn months (September, October, November); this can be seen in the maps as well as the graphs.

Figure 10: Correlation of the gridded soil moisture products as compared to in-situ station observations for the full year as well as seasons for the US. Subdivided in consecutive 4-year periods as well as for the longest period that data is available. Whiskers show the median and the IQR. The number below the whiskers are the number of years considered and the number above is the number of observations used. 

Figure 11: Correlation of the gridded soil moisture products as compared to in-situ station observations for the full year and different three monthly periods for consecutive 4 year periods as well as for the longest period that data is available.

2.1.9. Comparison to previous dataset versions

A comparison of the correlation between the soil moisture dataset and in-situ observations is provided in Figure 12 for different versions of the CCI product (including v04.2 upon which the C3S product v201812 is based). It can be seen that on average the correlation between the in-situ and satellite derived soil moisture datasets improves with the development of the product.

In the comparison the correlations are shown for different periods. This figure shows that the correlation between the products and the ISMN data is better for the later periods which is likely due to the increased observations available as input to the CCI products within this time period.

Further information on the assessment of previous CCI versions against reference data is available. Dorigo and Gruber (2015) provides an extensive evaluation against v0.1, employing all useable observations from the ISMN (Dorigo et al., 2017a). Another assessment (Albergel et al., 2013) also considered v0.1 but in relation to the ERA-Interim / Land dataset. Version 2.2 has also been subject to validation against in-situ observations (Fang et al., 2016).

In general, the products were deemed to agree well with in-situ observations, but lack behind the performance of those obtained for LSM simulation integrating observed precipitation; this may be due to the discrepancy between the installation depth of the in-situ probes (typically 5 cm) and the typical depth of ~2 cm represented by the satellite-derived datasets (Dorigo et al., 2017a).

Figure 12: Correlation between the CCI product (different versions) and ISMN data. Includes v04.2 upon which C3S v201812 is based. The results are shown for all seasons, for different three-year periods between 1997 and 2016 (first 5 panels) and in the final panel for the entire time period of the each product.

To demonstrate the differences between the previous C3S version (v201706) and a prototype version of the current dataset (v201806), a summary of the comparisons of the dataset to the ISMN data is shown in Table 4.

The summary of the results show that the comparison to ISMN for the overall mean metrics (not split by land cover or climate classes) is very similar (to 3 d.p.) for the different dataset versions for the case of both the latest merging period and the complete data period. However, there is a slight improvement in the metrics for the PASSIVE and COMBINED products in the newer version.

Table 4: Results of comparison against ISMN for different C3S dataset versions.

(1) For all comparisons, v20180830 of the ISMN dataset is used (see Section 2.1.2 for further details).
(2) For ACTIVE this is the ASCAT period (2007-01-01 to 2017-06-30); for PASSIVE this is the SMOS / AMSR2 period (2012-07-01 to 2017-06-30); for COMBINED this is the ASCAT / SMOS / AMSR2 period (2012-07-01 to 2017-06-30).
(3) For ACTIVE this is 1991-01-01 to 2017-06-30 and for PASSIVE and COMBINED, this is 1978-11-01 to 2017-06-30.

Figure 13: Correlation (Pearson) for C3S v201706 (left) and C3S v201806 (right) with ISMN (0 – 5 cm sensor depth) for the C3S COMBINED product split by land cover classes. Comparison is for the time period 2012-07-01 to 2017-06-30 (ASCAT, SMOS and AMSR2 period for the period when both versions of the dataset have observations). The boxplots show the mean value and interquartile range. 

Figure 14: ubRMSD for C3S v201706 (left) and C3S v201806 (right) with ISMN (0 – 5 cm sensor depth) for the C3S COMBINED product split by land cover classes. Comparison is for the time period 2012-07-01 to 2017-06-30 (ASCAT, SMOS and AMSR2 period for the period when both versions of the dataset have observations). The boxplots show the mean value and interquartile range.

2.2. Accuracy – Comparison against Land Surface Models

2.2.1. GLDAS v2.1

C3S v201812 has been compared against GLDAS v2.1 (from 2000-01-01 to 2018-12-31 (the end of the C3S product)). The spatial distribution of the correlation (Pearson) and the ubRMSD are shown in Figure 15 and Figure 16 respectively. This comparison has been generated using the QA4SM service. Note that the GLDAS data is masked in this case for snow covered conditions.

Figure 15 shows that the correlation between GLDAS v2.1 and the COMBINED product show expected spatial patterns. There is a high positive correlation between the products in most temperate zones, with low and near-negative correlations being most prevalent in the northern, boreal regions. The deserts show little to no positive or negative correlation. It is noted that this is an improvement over previous presented comparisons to GLDAS v2.1; this is due both to the improved algorithm of the C3S product and the masking of snow covered areas when undertaking the comparison to GLDAS (using data provided in the GLDAS dataset).

Figure 16 shows that the ubRMSD between GLDAS v2.1 and the COMBINED product is low in most regions, with a significant portion being below the 0.04 threshold required in the KPIs (see Section 4). Again, there appears to be a lower agreement between the products in the boreal regions, as is apparent in the correlation map (Figure 15).

Figure 15: Correlation (Pearson's) of the C3S v201812 COMBINED product with GLDAS v2.1 (covering the time period 2000-01-01 to 2018-12-31). 

Figure 16: ubRMSD of the C3S v201812 COMBINED product with GLDAS v2.1 (covering the time period 2000-01-01 to 2018-12-31. The threshold for the KPI (0.1 m³ / m³) is shown as red. 

To compare the results for the current and previous version (v201812 vs. v201706) a comparison of the results for the GLDAS comparison for correlation (Figure 17) and the ubRMSD (Figure 18) has been undertaken.

Correlation improvements are shown in blue. In general, the correlation has improved in most forested regions, with some degradation in the correlation in isolated areas (see Canada and Siberia). In the deserts, particularly the Sahara, the correlation has improved in most areas.

For ubRMSD improvements are also shown in blue. The spatial patterns of the ubRMSD difference as quite different to the correlation. A larger area of degradation can be seen in both Canada and Russia as well as in South America. Improvements can be seen globally in ubRMSD. However, the magnitude of all changes are very small (10^-3) therefore, although these differences to the correlation pattern are apparent, they are not considered significant.

Further analysis against GLDAS has been undertaken in the study of the most recent merging periods for the ACTIVE, PASSIVE and COMBINED products in section 7 to determine the impact of introducing the new GLDAS v2.1 into the product (rather than the previously used v1).

Figure 17: Difference between the correlation for C3S v201812 and GLDAS v2.1 and C3S v201706 and GLDAS v2.1. (v201812 correlation minus v201706 correlation; blue represents an improvement in the correlation). 

Figure 18: Difference between the ubRMSD for C3S v201812 and GLDAS v2.1 and C3S v201706 and GLDAS v2.1. (v201812 ubRMSD minus v201706 ubRMSD; blue represents an improvement in the ubRSMD).

2.2.2. ERA5

The C3S v201812 product has been compared to the newly available (since January 2019) ERA5 product¹⁰. ERA5 covers the period 1979-01-01 to within 3 months of present day and provides soil moisture data for different depth layers. Here "swvl1" is used which represents the first 7 cm of the soil. The correlation (Pearson's) (Figure 19) and ubRMSD (Figure 20) between C3S v201812 and ERA5 are presented.

The correlation (Figure 19) shows expected patterns which are similar to those shown for GLDAS v2.1 (see Figure 15), the exception being the larger area showing negative correlations in the North. This is due to snow / ice covered conditions not being flagged out in the ERA5 product (note that future assessments will address consistent flagging between comparisons). In general, high correlations are shown in most regions.

Figure 19: Correlation (Pearson's) of the C3S v201812 COMBINED product with ERA5 (covering the time period 1979-01-01 to 2018-12-31). 

The ubRMSD (Figure 20) shows a much higher value than expected in some areas, with the KPI threshold of 0.1 m³ / m³ being breached in the North. This is an unexpected pattern, especially considering the good performance of GLDAS v2.1 in this metric (see Figure 16). This may again be due to observations contaminated by snow and ice cover being used, as is the case for the correlation. Another factor may be that the GLDAS product performs better as this is used for scaling in the COMBINED product (that being assessed here) hence some of the characteristics of GLDAS are transferred to the C3S product.

Figure 20: ubRMSD of the C3S v201812 COMBINED product with ERA5 (covering the time period 1979-01-01 to 2018-12-31). 

To assess the differences between the current (v201812) and previous version (v201706) of the dataset, the results of the comparison against ERA5 for both products are compared for the correlation (Figure 21) and ubRMSD (Figure 22).

In Figure 21, the improvements in the correlation between the products are shown in blue, with worse performance shown in red. There appears to be some improvements in the product in the Sahara and the USA and at the most northern latitudes. However, there is a distinct reduction in performance in Scandinavia as well as in some desert regions. This is unexpected as comparisons against ISMN and GLDAS v2.1 have both shown improvements overall. These differences should be further investigated in continued development of the C3S product.

Figure 21: Difference between the ubRMSD for C3S v201812 and ERA5 and C3S v201706 and ERA5. (v201812 correlation minus v201706 correlation; blue represents an improvement in the correlation). 

Figure 22: Difference between the ubRMSD for C3S v201812 and ERA5 and C3S v201706 and ERA5. (v201812 ubRMSD minus v201706 ubRMSD; blue represents an improvement in the ubRMSD). 

It is known that there are discontinuities in the ERA5 product¹¹ which can be clearly seen in the example timeseries presented in Figure 23 for the lower depths. For example in the top timeseries, a break in layer 4 can be clearly seen in 2015. However, it should be noted that there is no discernible break in the timeseries for the upper most soil layer (swvl1), however, this assertion is made solely from visual interpretation of the timeseries. Further assessment should be undertaken to determine if this assertion holds true throughout the data product.

Figure 23: Example timeseries for volumetric soil water content in the ERA5 product with different depth layers shown in different colours. Two example locations are shown: (top) a desert site in Algeria and a desert site in Mauritania (bottom).

2.3. Stability – Trend monitoring

The methods for monitoring the stability of the dataset are still under development. Here, preliminary results are presented to demonstrate the methods developed thus far along with a discussion of how the methods will be developed for future versions of the product assessment. To monitor the stability of the product, two methods are being developed, both discussed below.

2.3.1. Accuracy evolution

The evolution of the ubRMSD between the ESA CCI SM v04.5 product and GLDAS v2.1 between 2000-01-01 and 2018-12-31 is used to assess how the accuracy of the product develops over time. The ubRMSD is calculated for each year and then the trend over time is calculated for every pixel (Theil-Sen estimator; median slope). The evolution of the ubRMSD globally is shown in Figure 24, with the global trend of the ubRMSD shown in Figure 25.

The evolution of the ubRMSD over time (Figure 24) shows that, in general, the product is achieving near to the KPI accuracy requirement of 0.04 m³ / m³ each year. There are some years where there is an improvement on this figure, for example, in 2003, however, this perceived improvement is likely due to reduced availability of data in this year (due to the reduced performance of the ERS sensor).

Figure 24: Evolution of the ubRMSD between ESA CCI SM v04.5 and GLDAS v2.1 over time. ubRMSD is calculated per year. Results presented are global. 

Figure 25: Trends in the ubRMSD between ESA CCI SM v04.5 and GLDAS v2.1 (from 2000-01-01 to 2018-12-31). The median slope of the Theil-Sen estimator is shown.  

The trends in the ubRMSD (Figure 25) can be interpreted thus: red shows an increase in ubRMSD which means worse performance of the product over time; blue shows a reduction in the ubRMSD which means better performance of the product over time; and white shows no change, which means the product accuracy remains stable over time. The most notable feature of this map is that, in the northern, highly vegetated areas, the performance of the product becomes much worse over time. In some more temperate regions, for example, the USA, Australia and India, the performance improves over time. The most stable regions are in dry areas, for example, parts of the Sahara, Australia and Southern Africa.

2.3.2. Dataset statistics evolution

The second method which is being investigated for monitoring stability is the consideration of the dataset statistics over time. Note that this method is very experimental, and work is ongoing to fully understand the information that maybe gained from this analysis.

In this method, the first step is the removal of the high frequency events from the dataset. This is done through the calculation of a 35-day moving average of the soil moisture. This "seasonality" is used is to reduce effect of random noise from the dataset including high frequency events such as rainfall. The variance of the seasonality is then calculated per year (note that future versions of this work will including looking the variance per season).

The evolution of the variance of the dataset over time is shown in Figure 26. This figure shows that the variance of the global dataset remains quite stable over time. However, there is a particular region between 1987 and 1991 which shows high variance in the variance; this coincides with the period where only SSM/I data is available. This indicates that in this period, the characteristics of the SSM/I sensor are changing.

As was the case with the accuracy monitoring method shown above, there are years shown in this method where the variance appears to reduce (which would indicate a better dataset). Again, this occurs in 2003, but it is thought this is likely due to reduced data availability in this year (due to the reduced performance of the ERS sensor).

Figure 26: Evolution of the ESA CCI SM v04.5 seasonality (35-day moving average) variance over time with the variance being calculated per year. Results presented are global. 

The trends in the variance of the dataset over time are shown in Figure 27. The trends presented here are very different from those shown in the previous method (i.e. in Figure 25). There is a tendency for the areas with the higher changes in variance to be nearer to the coastlines, perhaps indicating these pixels are contaminated by signal from open water. In addition, there is a clear increase in variance in areas close to tropical forests. Further interpretation of these results is required to understand the applicability of this method to stability monitoring.

Figure 27: Trends in the ESA CCI SM v04.5 seasonality (35-day moving average) variance (from 2000-01-01 to 2018-12-31). The median slope of the Theil-Sen estimator is shown.

2.4. Stability – Breakpoint detection

A procedure has been developed at TU Wien to test for potential inhomogeneities in the SM CDRs (Preimesberger et al.) Breaks may occur as a result of merging different sensor combinations over time, as shown in Figure 28. Such breakpoints may therefore appear between periods with different input sensors. Structural inhomogeneities may affect statistics such as trends and changes in extreme values (percentiles) and therefore should not only be detected but also corrected.

Figure 28: Potential break times in the ESA CCI SM product (version 04.4) corresponding to changeovers in the sensors.

Work to optimise the break-point detection has been ongoing; the use of non-parametric statistical tests (Fligner-Killeen test for homogeneity of variances and Wilcoxon rank-sums test for shifts in population mean ranks) against reference datasets has proved successful. The methods have been demonstrated using both in-situ measurements from the ISMN as well as from surface model simulations (MERRA2 SM) (Su et al., 2016) as a reference.

For C3S SM v201812, the breaks have been assessed using this method and the results are presented in Figure 29 for selected break times tested between 2002 and the end of the product (2018-12-31).

At all break times detected, there are significantly more mean breaks than variance breaks. The spatial patterns of the breaks vary between the break times, with Africa showing the most consistent set of locations where a break is detected.

The highest number of breaks appears to occur in 2007 which coincides with the introduction of the ASCAT-A sensor in to the timeseries. However, the introduction of AMSR2 in 2002 also seems to have a significant impact especially in Central Asia. In 2015, ASCAT-B is introduced in to the dataset (final panel in Figure 29); this appears to effect the dataset the most in central Australia, with both mean and variance breaks detected at this stage. This may indicate that the ASCAT-A and B products require further inter-calibration prior to ingestion. This will be investigated prior to the generation of the next product (v201912).

The observed breaks are likely to be caused by one of two elements of the processing algorithm:

• The scaling of the datasets to a common reference can result in unrealistic jumps in the timeseries.
• The error characterisation used to generate the merging weights for the product are fixed per location for each sensor period and does not take into account changes in the sensors over these periods. This is a product of how the triple collocation used to characterise the errors works.

To address the first point, an enhanced scaling method which considers the upper and lower percentiles in a better way will be implement. To address the second point, in future versions of the algorithm, a temporally evolving error characterisation will be investigated.

Figure 29: Breaks detected in the C3S v201812 product for selected break times from 2002 to the end of the product. Bright green shows variance breaks, whilst mean breaks are shown in red and a combination of the two are shown in blue. 

No adjustment of the breaks is undertaken for the current product. However, in the future, this will be implemented as part of the algorithm. To adjust detected breaks in the data set, three methods have been investigated:

1. LMP – Linear Regression Model Pair Matching: Differences in parameters of two linear regression models of data (one model from data prior to the break being tested and one model from data after the break being tested) used to derive corrections for SM observations before the break.
2. HOM – Higher Order Moments: A higher order polynomial regression model from observations after a break is used to create homogeneous predictions before the break. Corrections are derived from Locally Weighted Scatterplot Smoothing (LOWESS) fitted differences in quantiles of the observed and predicted ESA CCI SM values before the break. Quantiles are derived using L-moments statistics and KS testing.
3. QCM – Quantile Category Matching: Spline-fitted differences in empirical CDFs of the ESA CCI SM and reference SM (between quantile categories, i.e. average SM within a number of quantile ranges) values before, and after a break are used to find corrections for quantiles of ESA CCI SM before the break.

Adjustment is performed iteratively, with the goal that across each detected break, changes in means and variances are matched to follow changes within the reference data set (relative bias correction) and homogenised observation series (with respect to the reference data set) are derived. The results of the correction performed on v04.4 of the CCI dataset are shown in Figure 30 for each of the three methods.

Figure 30: Results of the inhomogeneity testing (between HSP₃ and HSP₄) before any correction methods have been applied (top) with the results of the testing after the correction methods are applied (for each method as indicated) (bottom). Adapted from Preimesberger et al. (In prep.).

2.5. Spatial and temporal completeness

It is important to consider the spatial and temporal completeness and consistency of the product as these can be a key deciding factor for the users in terms of whether or not the product is suitable for their application.

The spatial and temporal coverage of the product is presented (shown in Figure 31 and Figure 32) in terms of the number of valid (unflagged) observations available. The figures show that the coverage is better in Europe, South Africa and the continental US than some in other parts of the world as well as the improvement in the availability of data post-2007 as new sensors became available (see Figure 28 above for further details of sensor periods). This is as expected for the product due to the orbital paths of the satellites resulting in higher coverage in equatorial regions. The reduced coverage in boreal and tropical region is as expected due to the high Vegetation Optical Depth (VOD) expected in these areas. In addition, the coverage of the northern most latitudes in snow and ice for long periods also reduces the coverage in these areas.

Figure 31: Fractional coverage of the C3S COMBINED soil moisture product for the ASCAT / SMOS / AMSR2 period (2012-07-01 to 2018-12-31). Expressed as the total number of daily observations per time period divided by the number of days spanning that time period. 

Figure 32: Fraction of days per month with valid (i.e. unflagged) observations of soil moisture for each latitude and time period for the COMBINED product.

2.6. Timeseries analysis

Analysis of timeseries from a small number of locations provides an insight into the behaviour of the product for different climate and land cover types. Five points have been chosen for which ISMN in-situ data is available (and were used in the above assessment). Details of the points are provided in Table 5 and are shown on a global map in Figure 33.

Table 5: Details of locations chosen for time series analysis.

Note: all are classified as having 'medium' soil texture. 

Figure 33: Locations of the points used in the time series comparison (points are given at the C3S gpi location). 

The timeseries (temporally aggregated per month) for the individual locations for the ACTIVE, PASSIVE and COMBINED products are given in Figure 34. In general, the timeseries appear to follow expected seasonal cycles at each location, i.e. winters are wetter and summers drier and, in the case of gpi 890047 (which is located in Alaska), there are gaps in the data where the location is covered by snow each winter.

Of note for these timeseries, is the sudden drop in the values associated with the PASSIVE product around 2011 which can be seen in all but the cropland land cover type. Whilst this is not so pronounced for the grassland, it is clear in both the sparse vegetation and broadleaf forest. This drop coincides with the drop out of AMSR-E. It is noted that the COMBINED product does not suffer such a marked changed despite including the same sensors; this difference may be due to the scaling of the data to GLDAS v2.1 or due to the inclusion of the active sensors in this time period. Further information on this aspect is provided at section 5. 

Figure 34: Time series comparison for the COMBINED, ACTIVE and PASSIVE products of C3S v201812 for the gpis and land cover types stated for each plot. Note: here, the ACTIVE product is divided by 100 to allow it to be plotted on the same axis as the other products.

2.7. Uncertainty analysis

The algorithm used to develop the C3S soil moisture product utilises triple collocation analysis to generate weightings for the combination of different soil moisture observations (Gruber et al., 2017). The SNR calculated as part of the triple collocation process is used to weight the sensors within that merging period. In combination with error propagation techniques, a per-pixel uncertainty is provided within the C3S soil moisture product in the "sm_uncertainty" field.

The weights used within the ASCAT / SMOS / AMSR2 period is shown in Figure 35. These weightings show that ASCAT performs best in highly vegetated areas such as the boreal forests in the north whereas AMSR2 performs best in dry regions.

Figure 35: Weightings used for merging ASCAT (red) / SMOS (blue) / AMSR2 (green) within the C3S v201812 product. These maps are spatially gap filled by regression analysis between the signal-to-noise ratio used in the generation of the weights and vegetation optical depth (VOD).

The evolution of the 'sm_uncertainty' field per latitude over the duration of the C3S v201812 product is shown in Figure 36. This figure shows that the uncertainty associated with the product reduces over time. This indicates that in sensor periods closer to the present, the original products are much closer together in absolute values than in the sensor periods nearer the start of the C3S product. Throughout the product time period, the uncertainties are always higher at latitudes where there is higher vegetation cover, for example at 10 degrees south. This is as expected as soil moisture is harder to retrieve in these areas and there is higher variance in the product where this is the case. 

Figure 36: Monthly averages of the uncertainty associated with the C3S v201812 COMBINED product per latitude over time.

3. Application(s) specific assessments

Currently, no application(s) specific assessments have been undertaken for the December 2018 version (v201812) of the C3S soil moisture dataset. However, to demonstrate the usefulness of the product, we present here a couple of examples of the previous product (v201706) capturing particular events within the past year.

3.1. State of the Climate 2018

The C3S data has been used in the 2018 State of the Climate report produced by ECMWF¹² . In the report, the C3S SM v201812 PASSIVE soil moisture anomaly data is compared against the ERA5 soil moisture data (Figure 37). The trends shown match well with ERA5. 

Figure 37: Annual soil moisture anomaly for 2018. From the European State of the Climate report.

3.2. Mozambique Cyclone

On March 14^th 2019, cyclone Idai made landfall in Mozambique causing several hundred deaths¹³ . The event was captured within the C3S v201706 COMBINED soil moisture anomalies (dekadal). In Figure 38, the cyclone event is shown as wetter than normal conditions in the right-hand side image. 

Figure 38: Cyclone Idai (2019-03-14) captured in the dekadal soil moisture anomaly data for C3S v201706 COMBINED. On the left, the dekad prior to the cyclone is shown and on the right, the dekad during which the cyclone occurred. The area affected is circled in yellow on the right-hand side image.

4. Compliance with user requirements

The requirements for the C3S soil moisture product are a set of KPIs defined from consideration of user and GCOS requirements. The KPIs are shown in Table 6.

Table 6: Key Performance Indicators (KPIs) for the C3S Soil Moisture Product

An independent accuracy assessment has been conducted comparing the product against in-situ observations with different conditions taken into account, i.e. soil depth, land cover etc. In general, there is a variability in the correlation between the datasets, with correlations ranging from between 0.4 to above 0.8 depending on the conditions and the locations of the in-situ stations used. The ubRMSD, which can be directly taken as a measure of accuracy, is demonstrated to be below 0.10 m³ / m³ for all of the different conditions analysed, including land cover type (currently grassland and tree cover are considered). Therefore, the minimum threshold KPI for the C3S product for accuracy has been met globally. However, the GCOS target requirement of 0.04 m³/m³ is only met under certain land cover (grassland) and climate (semi-arid) conditions. Notice, however that the use of in-situ data for satellite validation is complicated by the presence of representativeness errors, which inflates the actual errors. The effect of representativeness will be addressed in future quality assurance activities, even as a finer categorisation of the product skill according to land cover type or vegetation cover.

A comparison against the LSMs GLDAS v2.1 and ERA5 has also been undertaken to provide a wider global view of the product. The comparison against GLDAS v2.1 has shown expected results, with the KPI threshold of 0.1 m³ / m³ met in most regions (the exception being areas with high topographic complexity). However, for the ERA5 comparison there are several unexpected spatial patterns and, in many areas, the KPI threshold target is not met. Further investigation is needed to ascertain the source of these discrepancies; it is possible these are a result of insufficient masking of highly vegetated areas and frozen soils / snow conditions within the comparison process.

The stability of the C3S product has been assessed in terms of the change in accuracy (when compared to the GLDAS v2.1 product in the period 2000-01-01 to 2018-12-31). The accuracy between the products (ubRMSD) has been calculated per year and the trends in the accuracy calculated. Spatially, the assessment shows that there are some significant changes in the accuracy in the northern boreal areas, with the most stable regions being arid areas. The KPI threshold for stability of 0.05 m³ / m³ / y is met when assessed using this method for most geographical regions. Another stability assessment method using the evolution of the standard deviation of the product over time is also being investigated, however, this is not yet mature enough to draw conclusions on the compliance with the user requirements.

5. Outcomes of the ACTIVE and PASSIVE quality control

5.1. Introduction

The results of the validation of C3S v201812 ACTIVE and PASSIVE products are not provided in this report, however two outcomes of the validation are discussed in detail. During the quality control of the product and the associated ESA CCI SM v04.4, the following were noted:

• There are strong negative trends present in the PASSIVE product which appear to be unrealistic.
• The ASCAT data is subject to wetting trends over Europe in particular during the ASCAT period; these trends are not present in the COMBINED or PASSIVE products.

These issues will be addressed as part of the algorithm development in the CCI+ project. It is expected that these issues will be better understood and addressed where possible.

5.2. Strong negative trends in the PASSIVE product

Quality control on ESA CCI SM v04.5 (equivalent to C3S v201812) revealed that in some locations, the PASSIVE product shows strong negative trends which are not apparent in the COMBINED or ACTIVE products (see Figure 39). The timeseries show that after approximately 2010, there is a strong dip in values in the PASSIVE product. 

Figure 39: Monthly aggregated timeseries (time period 2000-01-01 to 2018-06-30) for selected locations in the ESA CCI SM v04.5 product for the PASSIVE and COMBINED products. Strong negative trends can be seen in the PASSIVE product after 2010 which are not seen in the COMBINED product. 

To identify the extent to which this seemingly unrealistic negative trend occurs within the product, a comparison of the trends in the COMBINED and PASSIVE products has been undertaken (see Figure 40) for the period 2007-01-01 to 2018-12-31. Note that some differences in the trends are expected, particularly in this period due to the potential issues identified with the ASCAT data stream (see Section 5.3). It is shown in the PASSIVE dataset (bottom panel of Figure 40) that there are several areas where the negative trend is particularly strong, for example in Brazil, Europe and China. It is expected that these areas are suffering from the same issue shown in Figure 39. This will be further investigated at the development of the next product version under the CCI+ programme. 

Figure 40: Comparison of the Theil-Sen trends (median slope) in the ESA CCI SM v04.5 (same algorithm as C3S v201812) COMBINED (top) and PASSIVE (bottom) products for the period 2007-01-01 to 2018-12-31. Note: different scales are shown on these maps such that the colours shown in each case are similar to one another; this is a result of the different absolute values provided in each data product.

5.3. Wetting trends in the ASCAT data

A recent assessment of the soil moisture anomalies (with respect to a climatological period of 1991 – 2010) have shown that the COMBINED, ACTIVE and PASSIVE products show different spatial patterns as well as the direction of the anomalies in recent years over Europe (see Figure 41).

The most striking difference is between the ACTIVE and PASSIVE products in most areas of Europe (with the exception of Scandinavia). Figure 41 shows Europe in May 2018; it is known that, at this time, the UK and France in particular were experiencing drier than normal conditions, therefore, it is expected that the PASSIVE dataset showing accurate anomalies.

Figure 41: Anomalies for the ESA CCI SM v04.5 COMBINED (left), ACTIVE (middle) and PASSIVE (right) products for May 2018 over Europe (climatological reference period of 1991-2010). The anomalies vary both spatially and in direction between the products. Note: different scales are shown on these maps such that the colours shown in each case are similar to one another; this is a result of the different absolute values provided in each data product.

This issue was first noticed over Europe; further investigation of global differences (see Figure 42) show that the differences do not persist as strongly in other geographical regions. Therefore, region-specific factors will be investigated as the source of these differences between the products. This will include considering the impact of Radio Frequency Interference (RFI) on the ACTIVE data as it is known that there has been a slight positive trend in RFI, particularly over Eastern Europe during the time period in which ASCAT data is used (Ticconi et al., 2017). However, it is also noted that the issue seems most prominent in Spring months and therefore, may be related to the vegetation correct used in the ASCAT product.

Figure 42: Anomalies for the ESA CCI SM v04.5 (same algorithm as C3S v201812) COMBINED (top) and ACTIVE (bottom) products globally for May 2018 (climatological reference period of 1991-2010). Note: different scales are shown on these maps such that the colours shown in each case are similar to one another; this is a result of the different absolute values provided in each data product.

6. Detailed comparison of C3S v201812 against C3S v201706

6.1. Introduction

The v201812 dataset is an updated algorithm over the previous versions (v201706 / v201801). A summary of the changes can be found in the ATBD (Dorigo et al., 2019b). This section provides a detailed comparison of the newest dataset version (v201812) against the previous version (v201706). The aim is to determine both that the dataset has been made to specification and to highlight any differences between the products which may of interest to users when using the products in their own applications.

In this section, the following products are compared:

• C3S v201706 (based on the ESA CCI SM v03.2 algorithm)
• C3S v201812 (most recent product, based on the ESA CCI SM v04.4 algorithm (Dorigo et al., 2019b))

6.2. Comparison of data coverage

The number of valid observations available in each product version has been compared for the ACTIVE, PASSIVE and COMBINED products. Both the COMBINED and the ACTIVE products show very little difference in terms of coverage; the most notable point being the increase in data in the COMBINED dataset in later periods both in the Arctic and between 5 and 10 degrees South. Therefore, these figures are not shown here.

The most important factor revealed by this assessment is the loss of data within the AMSR2 period within the PASSIVE product (see Figure 43). This is unexpected and is a mistake in the dataset. 

Figure 43: Difference between the data coverage (fraction of valid observations) for C3S v201706 and C3S v201812 for the PASSIVE product for the entire timeseries. Positive values indicate more data is available within the v201812 product.

6.3. Comparison of timeseries

The timeseries for the different products is compared here for the COMBINED product (see Figure 44). Details of the locations chosen are given in Section 2.6. Overall, the products appear to be similar at these locations, with the only slight difference being the absolute values of the COMBINED product: v201706 shows slightly lower values for most locations (with the exception of the grassland which shows differing behaviour over time). In general, the data gaps shown in the timeseries are similar, however, in the grassland, there appears to be more values towards the end of the time period in the newer product.

Although not shown here, a comparison of the different product versions has also been undertaken for the PASSIVE product. This comparison shows that the drop in values around 2011 were also present in the previous product version (see section 5 for further information). 

Figure 44: Time series for the different land cover classes considered. Showing the data for the COMBINED product from v201706 and v201812 aggregated to monthly time steps.

6.4. Comparison of daily images

Daily images for 2017-06-21 for each of the ACTIVE, PASSIVE and COMBINED products have been compared for C3S v201806 and v201706 (bias between them). The COMBINED product only is shown in Figure 45 (both the ACTIVE and PASSIVE products shows very little difference between the versions). Figure 45 shows that there are some large differences between the products; the newer product has higher soil moisture values in general. This is not considered an issue; the newer product is scaled to the new GLDAS version (v2.1 rather than v1.0) and it can be seen from Figure 47 in section 7 that this newer GLDAS version has higher values than the previous version, therefore, this change in the product is expected. 

Figure 45: Difference between C3S v201812 and v201706 for the daily COMBINED product for 2017-06-21.

6.5. Comparison of global statistics

To demonstrate the differences between the previous C3S version (v201706) and a prototype version of the current dataset (v201806), the global statistics have been computed for each dataset version and are provided in Table 7. These are for the latest merging period of each product, however only the statistics up to 2017-06-30 have been used (to ensure a reliable comparison of the statistics).

The statistics for the latest merging period show very little difference between the data product versions and indicate that the product is, overall, quite similar. However, as can be seen from the timeseries presented above (see Section 6.3) it is known that higher values are expected in the newer dataset than the older one. Therefore, the exercise was repeated for the entire timeseries for each product version and the results analysed. The results are consistent with those presented for the latest merging period – overall the statistics are similar for each product version.

Table 7: Dataset statistics for the different C3S versions TCDRs for the latest merging period for each product for the common time period. The numbers given are the mean values across all gpis in the dataset, i.e. the mean of the timeseries for one gpi is calculated and then the mean is taken from all gpis.

7. Assessment of the impact of new data streams

7.1. Introduction

The v201812 dataset is an updated algorithm over the previous versions (v201706 / v201801). A summary of the changes can be found in the ATBD (Dorigo et al., 2019b). However, in addition to the changes in the algorithm, there have been changes to the input data streams used. The impact of these changes are assessed in this section.

The changes to the data streams between the product versions are:

• Previously, GLDAS v1 (Rodell et al., 2004) has been used for scaling observations to a common reference to reduce systematic biases between the input sensor datasets and in the triple collocation process. As of the end of 2017, v1 is no longer available and therefore v2.1 is used in the C3S product as a replacement; this newer version will continue to be supported for several years. This change was first implemented in v201801 of the dataset.
• For AMSR2, SMOS and AMSRE, the Land Parameter Retrieval Model (LPRM) v6 is used (for the remainder of the passive sensors, LPRM v5 is used). While these are the same data streams as previously used, for AMSR2 and SMOS, the flagging of high VOD areas has changed. Previously, the flag was set to 0.5 VOD for all bands for all sensors; in the new C3S version, the VOD threshold varies depending on the sensor and band based on the saturation point of VOD for that sensor / band combination. This change was first implemented in v201812 of the dataset.

7.2. GLDAS

7.2.1. Comparison of GLDAS v1 to v2.1

Figure 46 shows the correlation (Pearson) between GLDAS v2.1 and v1.0. The map shows that in some areas the correlation is very low (around 0) showing that the product has changed significantly between the products. Such a low agreement between the versions has the potential to significantly impact the C3S product. The spatial patterns of the correlation are similar (but not exactly the same) as the spatial patterns seen in the correlation between v201801 and v201706 (see Figure 49). 

Figure 46: Pearson correlation between GLDAS v2.1 and v1.0. Global map (top); histogram of correlation (bottom left) and summary statistics (bottom right) (Note: the results are not masked for high VOD or snow). 

Figure 47 shows the bias between the GLDAS v2.1 and v1.0. There are clearly some large differences between the versions (supporting that seen in the correlation (Figure 46)). The most significant feature of this map is that the spatial patterns almost entirely match the spatial patterns seen in the bias between C3S v201801 and v201706 (see Figure 50). The correlation (Pearson) between the biases from each case is 0.90 indicating that the spatial patterns in the biases are similar. 

Figure 47: Bias between GLDAS v2.1 and v1.0 (v2.1 – v1.0). Global map (top); histogram of correlation (bottom left) and summary statistics (bottom right) (Note: the results are not masked for high VOD or snow).

Figure 48 shows the absolute bias between the GLDAS v2.1 and v1.0 datasets over the whole time period per latitude. It can be seen the largest differences are in the high northern latitudes and around 10 degrees north. 

Figure 48: Hovmöeller diagram of the bias between GLDAS v2.1 and v1.0

7.2.2. Impact of using GLDAS v2.1

Note the assessment here is undertaken using versions v201706 and v201801. The algorithm used to produce these datasets is the same, the only difference is the use of GLDAS v2.1 which was first introduced at v201801, hence the use of v201801 in this assessment.

Figure 49 shows the global correlation (Pearson) between the datasets. The correlation has been calculated for the time series of each point location in the overlapping time period for the datasets (i.e. data from 1978-11-01 to 2017-06-30 is used).

In general, the correlation is high with the majority of points providing a correlation of close to 1. The main areas where the correlation is lower are in the Sahara, in the high northern latitudes and in mountainous areas. 

Figure 49: Pearson correlation between v201801 and v201706 daily COMBINED products. Global map (top); histogram of correlation (bottom left) and summary statistics (bottom right) (Note: the results are not masked for high VOD or snow). 

Figure 50 shows the absolute bias between the two datasets (v201801 – v201706). There appears to be a slight bias over many parts of the globe, with higher negative biases seen in the Sahara, high northern latitudes and in mountainous regions (consistent with the correlation results shown in Figure 49. The mean bias appears to be low (0.015) however, this is approximately 6.70 %¹⁴ (using the mean global value for soil moisture for v201801). 

Figure 50: Bias between v201801 and v201706 (v201801 – v201706). Global map (top); histogram of correlation (bottom left) and summary statistics (bottom right) (Note: the results are not masked for high VOD or snow). 

7.3. Impact of new flagging in LPRM v6

7.3.1. Differences in the LPRM flagging

In all versions of the C3S dataset produced so far (v201706, v201801 and v201806), LPRM v6 has been used for SMOS, AMSR2 and AMSRE in the generation of the PASSIVE and COMBINED products. However, in v201706 and v201801 (based on ESA CCI SM v03.2), the flagging of high VOD used was different to the flagging used in the most recent version, v201806 (based on ESA CCI SM v04.3).

Previously, the flagging was set to 0.5 VOD for all sensors, for all bands. In the most recent version, the flagging is set based on the thresholds given in Table 8. These are based on an assessment of the saturation points of each sensor / band. It is assumed that where the data becomes saturated, the signal is 100% due to vegetation cover, rather than soil moisture. The flagging is applied by taking the monthly mean for each band and if the threshold is breached, the entire month is flagged as high VOD (and hence excluded from the final product).

Table 8: Thresholds used in the flagging of the passive sensors for high VOD.

These changes have a significant impact on the data coverage available for each sensor. Figure 51 shows the impact for AMSR2 6.9 GHz band. The change in the threshold results in additional data primarily at the edges of the boreal and rainforest regions where the VOD values are between 0.50 and 0.60. 

Figure 51: Impact of changing the high VOD threshold for AMSR2 6.9 GHz band from 0.50 to 0.60 in terms of spatial coverage. Map shown here is an average of the entire timeseries; thresholds are applied to monthly averages so coverage will change over the time period of the product. 

7.3.2. Impact on data coverage

The significant difference in the final dataset due to the change in the VOD threshold is the increase in data coverage. Figure 52 shows the difference between the data coverage of the product (fraction of valid observations per month) for a product which used the 0.5 VOD threshold for AMSR2 and SMOS (v201801) and that which used the newer thresholds given in Table 8 (v201806). The maps show the difference for the AMSR2 / SMOS period only (2012-01-01 to 2017-12-31)¹⁵.

The figure shows that there is some significant increases in the data coverage in forested European regions and in the boreal regions as well as in North America. However, unexpectedly, there is a severe reduction in data in both the Sahara and in eastern Asia. 

Figure 52: Comparison of the data coverage (fraction of valid observations per month) of the C3S PASSIVE product between v201801 and v201806 for the SMOS / AMSR2 period (2012-01-01 to 2017-12-31).

7.4. Summary of Findings

There are clearly differences between the new and old versions of the C3S dataset due to the introduction of the new GLDAS dataset. This has resulted in an overall bias between the datasets of approximately 7 %. The analysis of the GLDAS data (version 2.1 versus version 1.0) have clearly shown spatial patterns in the biases and correlation similar to those shown in the analysis of the C3S data, with the correlation of the biases of the two analysis being 0.90. Therefore, it is concluded that the differences in the C3S data are primarily related to the inclusion of the new GLDAS data version.

The LPRM data has significantly impacted the data coverage in the PASSIVE and COMBINED product for the latest merging periods. However, it appears there have been some data losses which is unexpected.

References

(JCGM), J. C. F. G. I. M. 2008. International vocabulary of metrology — Basic and general concepts and associated terms (VIM). VIM3: International Vocabulary of Metrology, 3, 104.

ALBERGEL, C., DORIGO, W., REICHLE, R. H., BALSAMO, G., DE ROSNAY, P., MUÑOZ-SABATER, J., ISAKSEN, L., DE JEU, R. & WAGNER, W. 2013. Skill and Global Trend Analysis of Soil Moisture from Reanalyses and Microwave Remote Sensing. Journal of Hydrometeorology, 14, 1259-1277.

BSI 2015. BSI Standards Publication BS EN ISO 9000:2015 Quality management systems Fundamentals and vocabulary.

DE JEU, R., VAN DER SCHALIE, R., PAULIK, C., DORIGO, W., SCANLON, T., KIDD, R. & REIMER, C. 2019a. C3S Algorithm Theoretical Basis Document (ATBD): Soil Moisture (v201812).

DE JEU, R., VAN DER SCHALIE, R., PAULIK, C., DORIGO, W., SCANLON, T., KIDD, R. & REIMER, C. 2019b. C3S Algorithm Theoretical Basis Document: Soil Moisture.

DORIGO, W., SCANLON, T., BUTTINGER, P., PAULIK, C. & KIDD, R. 2019a. C3S Product User Guide and Specification (PUGS): Soil Moisture.

DORIGO, W., SCANLON, T., JEU, R. D., SCHALIE, R. V. D., PAULIK, C., KIDD, R. & REIMER, C. 2019b. Soil Moisture Products from active and passive microwave sensors: Algorithm Theoretical Basis Document.

DORIGO, W., SCANLON, T., PREIMESBERGER, W., BUTTINGER, P. & KIDD, R. 2019c. C3S Product Quality Assurance Document (PQAD): Soil Moisture (v201812).

DORIGO, W., SCANLON, T., PREIMESBERGER, W., BUTTINGER, P. & KIDD, R. 2019d. Product Quality Assurance Document: Land Service: Soil Moisture ECV.

DORIGO, W., VAN OEVELEN, P., WAGNER, W., DRUSCH, M., MECKLENBURG, S., ROBOCK, A. & JACKSON, T. 2011. A new international network for in situ soil moisture data. Eos, 92, 141-142.

DORIGO, W., WAGNER, W., ALBERGEL, C., ALBRECHT, F., BALSAMO, G., BROCCA, L., CHUNG, D., ERTL, M., FORKEL, M., GRUBER, A., HAAS, E., HAMER, P. D., HIRSCHI, M., IKONEN, J., DE JEU, R., KIDD, R., LAHOZ, W., LIU, Y. Y., MIRALLES, D., MISTELBAUER, T., NICOLAI-SHAW, N., PARINUSSA, R., PRATOLA, C., REIMER, C., VAN DER SCHALIE, R., SENEVIRATNE, S. I., SMOLANDER, T. & LECOMTE, P. 2017a. ESA CCI Soil Moisture for improved Earth system understanding: State-of-the art and future directions. Remote Sensing of Environment.

DORIGO, W. A. & GRUBER, A. A. 2015. Evaluation of the ESA CCI soil moisture product using ground-based observations. Remote Sensing of Environment, 162, 380--395.

DORIGO, W. A., SCANLON, T. & CHUNG, D. 2017b. C3S Product Quality Assurance Document (PQAD): Soil Moisture.

DORIGO, W. A., XAVER, A., VREUGDENHIL, M., GRUBER, A., HEGYIOVÁ, A., SANCHIS-DUFAU, A. D., ZAMOJSKI, D., CORDES, C., WAGNER, W. & DRUSCH, M. 2013. Global Automated Quality Control of In Situ Soil Moisture Data from the International Soil Moisture Network. Vadose Zone Journal, 12, 0.

ENTEKHABI, D., REICHLE, R. H., KOSTER, R. D. & CROW, W. T. 2010. Performance Metrics for Soil Moisture Retrievals and Application Requirements. Journal of Hydrometeorology, 11, 832-840.

FANG, L., HAIN, C. R., ZHAN, X. & ANDERSON, M. C. 2016. An inter-comparison of soil moisture data products from satellite remote sensing and a land surface model. International Journal of Applied Earth Observation and Geoinformation, 48, 37--50.

GELARO, R., MCCARTY, W., SUÁREZ, M. J., TODLING, R., MOLOD, A., TAKACS, L., RANDLES, C. A., DARMENOV, A., BOSILOVICH, M. G., REICHLE, R., WARGAN, K., COY, L., CULLATHER, R., DRAPER, C., AKELLA, S., BUCHARD, V., CONATY, A., DA SILVA, A. M., GU, W., KIM, G.-K., KOSTER, R., LUCCHESI, R., MERKOVA, D., NIELSEN, J. E., PARTYKA, G., PAWSON, S., PUTMAN, W., RIENECKER, M., SCHUBERT, S. D., SIENKIEWICZ, M. & ZHAO, B. 2017. The Modern-Era Retrospective Analysis for Research and Applications, Version 2 (MERRA-2). Journal of Climate, 30, 5419-5454.

GRUBER, A., DORIGO, W. A., CROW, W., WAGNER, W. & MEMBER, S. 2017. Triple Collocation-Based Merging of Satellite Soil Moisture Retrievals. 1-13.

GRUBER, A., DORIGO, W. A., ZWIEBACK, S., XAVER, A. & WAGN, W. 2013. Characterizing Coarse-Scale Representativeness of in situ Soil Moisture Measurements from the International Soil Moisture Network. Vadose Zone Journal, 12.

LIU, Y. Y., DORIGO, W. A., PARINUSSA, R. M., DE JEU, R. A. M., WAGNER, W., MCCABE, M. F., EVANS, J. P. & VAN DIJK, A. I. J. M. 2012. Trend-preserving blending of passive and active microwave soil moisture retrievals. Remote Sensing of Environment, 123, 280-297.

PLUMMER, S., LECOMTE, P. & DOHERTY, M. 2017. The ESA Climate Change Initiative (CCI): A European contribution to the generation of the Global Climate Observing System. Remote Sensing of Environment.

PREIMESBERGER, W., SCANLON, T., SU, C. H., GRUBER, A. & DORIGO, W. In prep. Homogenization of struvtural breaks in a global multi-satellite soil moisture climate data record.

RIENECKER, M. M., SURAEZ, M. J., GELARO, R., TOLDING, R., BACMEISTER, J., LIU, E., BOSILOVICH, M. G., SCHURBERT, S. D., TAKACS, L., KIM, G.-K., BLOOM, S., CHEN, J., COLLINS, D., CONATY, A., SILVA, A. D., GU, W., JOINER, J., KOSTER, R. D., LUCCHESI, R., ANDERA, M., OWENS, T., PAWSON, S., PEGION, P., REDDER, C. R., REICHLE, R., ROBERTSON, F. R., RUDDICK, A. G., SIENKIEWICZ, M. & WOOLLEN, J. 2011. MERRA : NASA’s Modern-Era Retrospective Analysis for Research and Applications. Journal of Climate, 24, 3624-3648.

RODELL, M., HOUSER, P. R., JAMBOR, U., GOTTSCHALCK, J., MITCHELL, K., MENG, C.-J., ARSENAULT, K., COSGROVE, B., RADAKOVICH, J., BOSILOVICH, M., ENTIN, J. K., WALKER, J. P., LOHMANN, D. & TOLL, D. 2004. The Global Land Data Assimilation System. Bulletin of the American Meteorological Society, 85, 381-394.

SU, C. H., RYU, D., DORIGO, W., ZWIEBACK, S., GRUBER, A., ALBERGEL, C., REICHLE, R. H. & WAGNER, W. 2016. Homogeneity of a global multisatellite soil moisture climate data record. Geophysical Research Letters, 43, 11,245-11,252.

TICCONI, F., ANDERSON, C., FIGA-SALDANA, J., WILSON, J. J. W. & BAUCH, H. 2017. Analysis of Radio Frequency Interference in Metop ASCAT Backscatter Measurements. Ieee Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 10, 2360-2371.

WAGNER, W., DORIGO, W., DE JEU, R., FERNANDEZ, D., BENVENISTE, J., HAAS, E. & ERTL, M. 2012. Fusion of Active and Passive Microwave Observations To Create an Essential Climate Variable Data Record on Soil Moisture. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, I-7, 315--321.

WMO 2016. The Global Observing System for Climate (GCOS): Implementation Needs.

Related articles

• Page:
  SM 2020: Target Requirements and Gap Analysis Document (TRGAD)
• Page:
  SM v202012: Algorithm Theoretical Basis Document (ATBD)
• Page:
  SM v201912: Product Quality Assessment Report (PQAR)
• Page:
  SM v 201912: Algorithm Theoretical Basis Document (ATBD)
• Page:
  SM v201812: Algorithm Theoretical Basis Document (ATBD)