Last modified on Mar 18, 2024 14:58

Contributors: W. Dorigo (WD, TU Wien), T. Scanlon (TS, TU Wien), W. Preimesberger (WP, TU Wien), R. Kidd (RK, EODC)

Issued by: EODC/R. Kidd

Date: 16/06/2021

Ref: C3S_312b_Lot4_D2.SM.2-v3.0_202104_Product_Quality_Assessment_Report_i1.0

Official reference number service contract: 2018/C3S_312b_Lot4_EODC/SC2

History of modifications

Related documents

Acronyms

General definitions

Accuracy: The closeness of agreement between a measured quantity value and a true quantity value of a measure ((JCGM) 2008). The metrics used here to represent accuracy are correlation and unbiased Root Mean Square Difference (ubRMSD). These metrics are commonly used throughout the scientific community as measures of accuracy (Entekhabi et al. 2010).

Bias: Estimate of a systematic measurement error ((JCGM) 2008).

Error: Measured quantity value minus a reference quantity value ((JCGM) 2008).

Precision: Closeness of agreement between indications or measured quantity values obtained by replicate measurements on the same or similar objects under specified conditions ((JCGM) 2008).

Quality Assurance: Part of quality management focused on providing confidence that quality requirements will be fulfilled (BSI 2015).

Stability: Property of a measuring instrument whereby its metrological properties remain constant in time ((JCGM) 2008). Note that for earth observation activities, "measuring instrument" can be translated as "retrieved variable" and "metrological properties" can be interpreted as "variability of retrieved variable". Alternatively, the Global Climate Observing System (GCOS) (WMO 2016) defines stability as the extent to which the uncertainty of measurement remains constant in time; the GCOS requirements are stated as the maximum acceptable change in systematic error (usually per decade).

Scope of the document

The purpose of this document is to describe the results of the Quality Assessment (QA) for the soil moisture product developed by TU Wien, EODC and VanderSat for the Copernicus Climate Change Service (C3S). The product version assessed in this report is v202012.0.0, which was produced in February 2021.

The C3S soil moisture product suite provides PASSIVE, ACTIVE and COMBINED (passive + active) microwave soil moisture products on a daily, dekadal (10-days) and monthly basis. The data are provided in a regular 0.25 degree grid based on the World Geodetic System 1984 (WGS 84) reference system. The product is available globally between November 1978 and present day (for PASSIVE and COMBINED) and between 1991 and present day (for ACTIVE). For details about the products, we refer to the Product User Guide and Specification (PUGS) (RD1) (Dorigo et al. 2021a).

This document presents the results of QA activities that have been undertaken for the current Climate Data Record (CDR) dataset (v202012.0.0). The Intermediate Climate Data Record (ICDR) datasets are not currently assessed. However, note that, to achieve maximum consistency between CDR and ICDR, both products use the same Level 2 products (based on Near Real Time (NRT) data streams) and merging algorithms and thus have very similar quality characteristics.

A brief summary of the methodology used, described fully in the Product Quality Assurance Document (PQAD) (RD2) (Dorigo et al. 2021b), is provided. The results described here are primarily for the COMBINED daily product, however, an assessment of the ACTIVE and PASSIVE products was undertaken and the main findings of these assessments are presented in the report.

Executive summary

The purpose of the Product Quality Assessment Report (PQAR) is to describe the product QA results for the soil moisture product developed by TU Wien, EODC and VanderSat for the C3S service. The production of the product has been funded by C3S, a service which is managed by the European Centre for Medium Range Weather Forecasting (ECMWF) on behalf of the European Commission (EC). The product version assessed in this report is v202012.0.0, which was produced in February 2021.

The document presents the results of the quality assessments undertaken for the product, including the accuracy and stability assessment of the product. This document is applicable to the QA activities performed on the version of the CDR v202012.0.0. Currently, the assessment does not cover ICDRs but due to the high consistency between CDR and ICDR (both products use the same Level 2 products, based on NRT data streams, and merging algorithms), the QA assessment of the last years of the CDR can be readily transferred to the ICDR.

The QA results broadly include the following parts: accuracy assessment, stability assessment, demonstration of uncertainty estimates, comparison to previous versions of the product, and a completeness / consistency assessment. The first two sections focus on demonstrating that the Key Performance Indicators (KPIs) set for the product are met. Note these KPIs take into account GCOS and user requirements for the product. In addition, in this version of the document, there is also a detailed assessment of the product against the previous version.

Accuracy Assessment: In general, there is a slight variability in the correlation between the datasets, with correlations ranging from 0.4 to above 0.8; depending on the conditions and the locations of the in situ stations used. The unbiased Root Mean Square Difference (ubRMSD), which can be directly taken as a measure of accuracy, is demonstrated to be below 0.10 m³ / m³ for all the different conditions analysed. Therefore, the KPIs for accuracy have been met for in situ observations. The global comparison against the Global Land Data Assimilation System (GLDAS) Noah v2.1 has shown expected results, with the KPI threshold of 0.1 m³ / m³ being met in most regions (the exception being areas with high topographic complexity). For the ECMWF Reanalysis (ERA) 5 and ERA5-Land, the KPI threshold target is not met in some areas with reduced data coverage in the sub-Arctic zone.

Stability Assessment: The stability of the C3S product has been assessed in terms of the change in accuracy (when compared to ISMN network measurements). The accuracy between the products (ubRMSD) has been calculated per year, as well as trends in the median yearly accuracy. The KPI threshold for stability of 0.05 m³ / m³ / y is met when assessed using this method for all tested locations.

Compared to the previous version (v201912) the main algorithmic updates include the addition of soil moisture observations from another passive sensor - Soil Moisture Active Passive (SMAP). An updated retrieval algorithm is now used for all passive sensors (LPRM v6), including ones that are no longer operational (see RD3). A break in the passive product, due to insufficient intercalibration of measurements from AMSR-E and AMSR2 has been resolved. Long-term analyses (such as for soil moisture anomalies) profit from this in particular.

A comparison to previous products has been provided. The assessment demonstrates that the correlation between the in situ and satellite-derived products is similar to previous versions. Improvements are found for some networks. Increases in the data coverage of the PASSIVE and COMBINED products are expected due to the addition of SMAP (in the last merging period) and due to the use of LPRM v6 (across merging periods before 2012-07).

The spatial and temporal coverage of the product has been presented in terms of the number of valid (unflagged) observations available. It is shown that the coverage is better in Europe, Southern Africa and the contiguous United States (US) than in some other parts of the world.

Further, detailed assessment of the product has been undertaken, in particular for the ACTIVE and PASSIVE products as well as a detailed comparison against the previous product version. These assessments show that the dataset was generated correctly.

However, some (previously known) issues remain in the current version:

• ACTIVE wetting trends: Artificial wetting trends in HSAF ASCAT SSM also affect the ACTIVE product of C3S SM after 2007 (and to some extent the COMBINED product). These trends are most likely caused by radio frequency interference (RFI) and appear especially in densely populated areas such as Europe, East Asia and parts of the United States. The trends, which are already present in ASCAT observed backscatter time series, are currently corrected in an experimental version of ASCAT SSM. Once there is an official release of HSAF ASCAT SSM – including the backscatter correction, and it is used within the European Space Agency (ESA) Climate Change Initiative (CCI) SM dataset (currently expected for v7), subsequent C3S versions can also include the corrected data.
• ERA5 assessment: The assessment against the ERA5 dataset shows – similar to results found for the previous version - ubRMSD higher than the KPI thresholds in a few areas. This indicates that there is need to further improve flagging strategies for observations in sub-Arctic areas (where frozen soils occur) as was recently done for ESA CCI SM (van der Vliet et al. 2020).
• Inconsistent long-term SM trends: Inconsistent long-term trends (1978/1991 – 2020) are found between the ACTIVE, PASSIVE and COMBINED products. While this can be partly related to differences in time range, data density and scaling references of the products, it still reveals that there is need to improve the long-term consistency of the product.

The following issues that were found in the previous version are resolved in the new release:

• PASSIVE time series drop: A significant drop in soil moisture values after 2011 for several land cover types was found in the PASSIVE product of previous versions (until v201912). This was due to the performed scaling, which was insufficient due to missing overlapping observations between AMSR-E (the PASSIVE scaling reference) and AMSR2. This issue has been accounted for by using data from the last 3 years of AMSR-E and the first 3 years of AMSR2 as input for scaling. This way the break is corrected. Potential issues with this approach could arise from extreme events in either 3-year subset or from trends during that time, which might be lost during the scaling process. Future inclusion of additional radiometer measurements spanning over the AMSR-E/AMSR-2 periods can further improve the consistency of the PASSIVE product.

1. Product validation methodology

1.1. Validated products

C3S soil moisture provides passive (named PASSIVE), active (ACTIVE) and passive + active (COMBINED) soil moisture products on a daily, dekadal (10-days) and monthly basis. The time periods, over which each sensor is used, are provided in Table 1. The data are provided in a regular 0.25 degree grid based on the WGS 84 reference system. The product is available globally between November 1978 and present day (for PASSIVE and COMBINED) and between 1991 and present day (for ACTIVE). The product has been produced by TU Wien, Vandersat, and EODC. The ASCAT SSM product mentioned in some parts of this report is developed and provided by H SAF.

Table 1: Blending periods for the soil moisture product (ACTIVE, PASSIVE and COMBINED)

The C3S soil moisture product is generated from a set of passive microwave radiometers and active microwave scatterometers.

Radiometers include the Scanning Multichannel Microwave Radiometer (SMMR), Special Sensor Microwave Imager (SSM/I), TRMM Microwave Imager (TMI), Advanced Microwave Scanning Radiometer-Earth Observing System (AMSR-E), WindSat Spaceborne Polarimetric Microwave Radiometer (WindSat), Advanced Microwave Scanning Radiometer 2 (AMSR2) , Soil Moisture and Ocean Salinity (SMOS) and Soil Moisture Active Passive (SMAP).

Scatterometer observations are collected by the Active Microwave Instrument - Windscat (AMI-WS) and ASCAT (Metop-A and Metop-B) sensors. The “ACTIVE product” and the “PASSIVE product” are created by fusing scatterometer and radiometer Level 2 soil moisture products respectively; the “COMBINED product” is created by fusing Level 2 soil moisture products from both sensor types. Data files are provided in NetCDF-4 classic format as daily, dekadal and monthly images and comply with CF-1.8 conventions¹.

A detailed description of the product generation of C3S v202012.0.0 is provided in the Algorithm Theoretical Basis Document (ATBD) (RD3) (de Jeu et al. 2021) with further information on the product given in the PUGS (RD1) (Dorigo et al. 2021a). The underlying algorithm is based on that used in the generation of the publicly released ESA CCI versions 5.3 which is described in relevant documents in Plummer et al. (2017), Wagner et al. (2012), Liu et al. (2012), and Dorigo et al. (2017a). In addition, detailed provenance traceability information can be found in the metadata of the product.

The C3S SM product comprises a long-term Climate Data Record (CDR) which runs from 1978 (PASSIVE and COMBINED) or from 1991 (ACTIVE) to December 2019. This CDR is updated every dekad (approximately every 10 days) in an appended dataset called an Intermediate Climate Data Record (ICDR). The theoretical algorithm and the processing implemented in the CDRs and ICDRs are exactly the same and the data provided are consistent between them.

This current document is applicable to the QA activities performed on the version of the CDR v202012.0.0 (produced in February 2021). The COMBINED daily product is the focus of the validation activities presented in this report. However, the ACTIVE and PASSIVE daily products have also been validated and a summary of the outcomes is provided in Annex A. Annex B compares in more detail v201912 to v202012. Annex C contains validation results for the dekadal and monthly products.

1.2. Description of validating datasets

A combination of in situ and global reference datasets are utilised to assess the quality of the C3S soil moisture product. A list of the datasets utilised is provided in Table 2 with further details provided in the PQAD (RD2) (Dorigo et al. 2021b).

Table 2: Datasets utilised in the assessment of the data product

1.3. Description of product validation methodology

1.3.1. Method Overview

The methodology used in the assessment of the soil moisture product is described in Dorigo et al. (2017b). The methodology, including details on the validated products and the validating datasets, are briefly described here in the relevant sections. A summary of the validation results for the previous dataset version (v201912) may be found in Dorigo et al. (2020).

The quality assessment includes the following:

1. Assessment against in situ observations from the ISMN (Section 2.1),
2. Assessment against Land Surface Models (LSMs) including GLDAS, ERA-5 and ERA5-Land (Section 2.2),
3. Stability analysis through monitoring of accuracy trends and dataset statistics (Section 2.3),
4. Assessment of the spatial and temporal completeness and consistency of the product (Section 2.4),
5. Analysis of time series at selected locations (Section 2.5), and,
6. Analysis of the uncertainty information provided with the dataset (Section 2.6).

Additional information is also provided in the Annexes of the report:

1. Annex A: The main body of the report considers mainly the COMBINED CDR. Therefore, a separate Annex provides information on the validation results of the ACTIVE and PASSIVE products, summarising the key findings from these activities.
2. Annex B: The main body of the report provides some comparison between the current C3S version (v202012) and the previous version (v201912). Annex B provides a more complete analysis to demonstrate that the data have been produced correctly and show any differences between the products.
3. Annex C: The main body of the report refers to the daily images provided through CDS. This section evaluates the aggregated product (monthly and 10-daily /dekadal), which can also be downloaded.

1.3.2. Quality Assurance for Soil Moisture

Some analysis presented in this report has been undertaken using software specifically designed for the C3S soil moisture project. However, in parallel, an online validation service - Quality Assurance for Soil Moisture (QA4SM, available at https://qa4sm.eu) - has been developed, which undertakes similar tasks and is used for most of the global validations.

The QA4SM service is an online validation service, which allows the traceable validation of satellite derived soil moisture products. Currently, supported datasets include C3S SM, SMAPL3 36km SM, the H SAF ASCAT SSM, the ESA CCI SM product and the SMOS IC product. The comparisons can be carried out against ISMN, GLDAS, ERA5 and ERA5-Land reference data. The service provides different options for the filtering of datasets and for scaling the datasets to each other. CDR v202012 of C3S SM is available online.

The QA4SM service is continuously under development and does not currently have all the features required to undertake a full validation of the C3S data as required for the validation activities presented in the current document. As the QA4SM service is being developed, the analysis presented in this report will be updated with results from this system at the generation of new product versions.

1.3.3. Planned enhancements to the method

This section briefly details enhancements to the methods envisaged for the future. They will potentially be implemented at the generation of new product versions.

Committed area mask

In future assessments, it is proposed that a committed area mask, akin to that used in the validation of the H SAF soil moisture project (see Figure 1), is introduced. This mask is used to better understand the performance of the product globally and is based on several criteria and thresholds.

Figure 1: Committed area used by the H SAF soil moisture project. The mask is based on a combination of vegetation optical depth thresholds, snow / ice masking and land cover types.

Consistent masking of the validation datasets

Currently, the masking of the datasets used in the validation is undertaken dependent on the flags available within the separate data products. For example, the GLDAS masking is done based on soil temperature fields in GLDAS. However, between the different validations there is no consistency in which locations / timestamps are masked. The introduction of consistent masking would help improve the inter-comparison of the validation results using data from different sources.

Assessment of the discontinuities in ERA5

It is known that there are discontinuities in the ERA5 soil moisture variables due to the splitting of the processing into chunks. The assessment presented here shows that these discontinuities appear to not affect the upper most soil layer and therefore, the assessment of the C3S product against these data is appropriate. However, this assertion has been made solely on the visual interpretation of the ERA5 time series and further analysis should be carried out to determine if this is the case.

2.  Validation results

2.1. Accuracy – Comparison against ISMN

2.1.1. Introduction

The C3S dataset has been compared to the ISMN dataset and the correlations and ubRMSD between the two datasets calculated.

In addition to an overall comparison processed using the QA4SM service (Section 2.1.3), the comparison is also undertaken for different attributes of the soil moisture data (provided as metadata within the ISMN dataset). These are: the sensor depth (Section 2.1.4), soil texture (Section 2.1.5), Köppen-Geiger climate classes (Section 2.1.6) and land cover (Section 2.1.7). For further information on the origin of these attributes, see Dorigo et al. (2011).

The aim of assessing the accuracy for different attributes is to demonstrate the performance of the product under different conditions and demonstrate that KPIs DX.1 (where X is equal to 1 – 6) are met. See the PQAD (RD2) (Dorigo et al. 2021b), Table 2, for the list of KPIs; repeated in this document at Table 6.

2.1.2. ISMN data and comparison settings

The ISMN data used in the assessments presented in this document were downloaded from the ISMN data portal⁴ on 2021-01-31 (hence this is referred to as v20210131). The full list of networks used in the assessment can be found in the PQAD (RD2) (Dorigo et al. 2021b). The dataset consists of up to 576 stations within 25 networks, as shown in Figure 2.

Figure 2: Potential ISMN networks and stations used for validation. Left: For soil moisture between 0 and 5 cm depth; 576 stations in up to 23 networks are considered in the validation process. Right: For soil moisture between 5 and 10 cm depth, 554 stations in up to 17 networks. Note that the station selection for validation may vary depending on the availability of C3S SM, the validation time period and the measurement time of each ISMN station.

Where there is more than one station available within a grid cell, a simple average of the station data is taken prior to comparison to the C3S time series for all assessments, except for the overall comparison. In the overall comparison (processed through the QA4SM service) no averaging of the ISMN time series is undertaken. Instead, a comparison is undertaken for each ISMN station against each nearest neighbour C3S GPI; hence in these results some GPIs are over-represented.

To calculate the metrics for each assessment, the settings summarised in Table 3 are used. In summary, the datasets are spatially and temporally matched, filtered for high quality observations and the C3S data are scaled to the ISMN data using mean – standard deviation scaling. The metrics are then calculated using observations in the period 1978-11-01 to 2020-12-31.

Table 3: Settings used in the assessment of the daily C3S soil moisture product against the ISMN

2.1.3. Overall Comparison

The comparison of C3S v202012 has been processed using the QA4SM method against ISMN v20210131. The global map in Figure 3 shows the ubRMSD for each ISMN station to the nearest C3S grid cell; correlation (Pearson's) is shown in Figure 4. These figures show the expected spatial patterns, with high correlations and low ubRMSD seen at most ISMN locations. COMBINED v202012 preforms slightly better in this comparison than v201912.

Results from the following validation run are accessible online through QA4SM⁶ and are archived on Zenodo: 10.5281/zenodo.4736927

Figure 3: ubRMSD between C3S v202012 COMBINED and ISMN v20210131 for soil depths of 0 – 5 cm (left) and comparison with C3S v201912 COMBINED (right).

Figure 4: Correlation (Pearson's) between C3S v202012 COMBINED and ISMN v20210131 for soil depths of 0 – 5 cm (left) and comparison with C3S v201912 COMBINED (right).

2.1.4. Soil depth

The correlation between the C3S dataset and the in situ datasets are presented in Figure 5 for two different surface soil moisture depths (0-5 cm and 5-10 cm).

The deeper sensors have a lower correlation and higher ubRMSD than the sensors at shallower depths. This is expected as the product represents the first few centimetres of the soil surface (approx.). However, it may also be attributed to the decreasing errors associated with the ISMN data at these deeper depths - noting that the relation seems to break at depths > 1 m (Gruber et al. 2013).

Figure 5: Correlation (left) and ubRMSD (right) between C3S v202012 COMBINED and ISMN for soil depths of 0 – 5 cm and 5 – 10 cm. The boxplots show the median and interquartile range.

2.1.5. Soil texture

The correlation and ubRMSD between the C3S dataset and the in situ datasets are presented in Figure 7 for the different soil textures (fine, medium and coarse; stratification provided from the ISMN dataset (Dorigo et al. 2011) and shown in Figure 6).

The product appears to perform best for medium texture soils in terms of correlation and ubRMSD (for both depth layers). The spread in ubRMSD for fine soil textures is higher than for medium texture. The coarse texture soils have very few observations available, therefore the results for this soil texture are not considered reliable.

Figure 6: Soil texture classification used for the ISMN comparison. The percentage clay, silt and sand are taken from the ISMN metadata which in turn is retrieved from the HWSD.

Figure 7: Pearson's R (left) and ubRMSD (right) between C3S v202012 COMBINED and ISMN for different soil texture classes in 0-5 cm (top) respectively 5-10 cm (bottom) depth. The boxplots show the median and interquartile range.

2.1.6. Köppen-Geiger climate classes

The correlation and ubRMSD between the C3S dataset and the in situ datasets are presented in Figure 8 for different Köppen-Geiger classes (BSx⁷, Csx / Dsx⁸ and Cfx / Dfx⁹), a global map of which is shown in Figure 9 (stratification provided from the ISMN metadata - see Dorigo et al. (2011)).

The correlation between the in situ measurements and the C3S product is similar across all Köppen-Geiger classes, with Csx / Dsx performing the best in terms of correlation and second best in terms of ubRMSD. The ubRMSD can be taken as a measure of accuracy; the graphs show that in this case, the mean value for all of the different climate classes are under the 0.1 m³ / m³ threshold set out in the KPIs; therefore the KPIs are achieved.

Figure 8: Correlation (Pearson's, left) and ubRMSD (right) between C3S v202012 COMBINED and ISMN for different climate classes in 0-5 cm (top) respectively 5-10 cm (bottom) depth. The boxplots show the median and interquartile range.

Figure 9: Köppen-Geiger classes. The classes used in this assessment are BSx, Csx / Dsx and Cfx / Dfx. The figure is taken from http://hanschen.org/koppen.

2.1.7. Landcover classes

The correlation between the C3S dataset and the in situ datasets are presented in Figure 10 for five different land cover classes (cropland, grassland, tree-cover, urban areas, other); stratification provided from the ISMN dataset (Dorigo et al. 2011).

The ubRMSD can be taken as a measure of accuracy and the KPIs specifically state the acceptable accuracy level of the product is between 0.01 and 0.1 m³ / m³ for different land cover types. The worst case ubRMSD shown here is under 0.1 m³ / m³ (median and IQR) for tree cover. Therefore, overall the KPIs are met.

Figure 10: Correlation and ubRMSD between C3S v202012 COMBINED and ISMN for different land cover classes. The boxplots show the median and interquartile range.

2.1.8. Comparison to previous versions

A comparison of the correlation between satellite SM and in situ observations is provided in Figure 11 for different versions of the CCI product (including v05.2 upon which the C3S product v202012 is based). It can be seen that on average the correlation between the in situ and satellite derived soil moisture datasets improves with the development of the product.

In the comparison, the correlations are shown for different periods. Figure 11 shows that the correlation between the products and the ISMN data are better for the later periods. This is likely due to the increased observations available as input to the CCI products within this time period and the good performance of SMAP. Especially the period after 2015 is positively affected by the addition of SMAP and shows higher correlations than the previous version (CCI v4, resp. C3S v201912).

Further information on the assessment of previous CCI versions against reference data is available. Dorigo and Gruber (2015) provides an extensive evaluation against v0.1, employing all useable observations from the ISMN (Dorigo et al. 2017a). Another assessment (Albergel et al. 2013) also considered version 0.1 but in relation to the ERA-Interim / Land dataset. Version 2.2 has also been subject to validation against in situ observations (Fang et al. 2016).

In general, the products were deemed to agree well with in situ observations, but lack behind the performance of those obtained for LSM simulation integrating observed precipitation, such as ERA5-Land. This may be due to the discrepancy between the installation depth of the in situ probes (typically 5 cm) and the typical depth of ~2 cm represented by the satellite-derived datasets (Dorigo et al. 2017a).

Figure 11: Correlation between the CCI product (different versions), ERA5-Land, ERA-Interim/Land and ISMN data. It includes v5.2 of ESA CCI SM upon which C3S v202012 is based. The results are shown for all seasons, for different three-year periods between 1997 and 2016 (first six panels) and in the final panel for the entire time period of each product.

To demonstrate the differences between the previous C3S version (v201912) and the current dataset (v202012), a summary of the comparisons of the dataset to the ISMN data is shown in Table 4. Overall, performance has improved for PASSIVE and COMBINED products over both the latest merging period and the complete data period. Performance has remained the same for the ACTIVE product.

Table 4: Results of comparison against ISMN (0-5 cm) for different C3S dataset versions (median values).

(1) For all comparisons, v20210131 of the ISMN dataset is used (see Section 2.1.2 for further details).

(2) For ACTIVE this is the ASCAT period (from 2007-01-01 onward); for PASSIVE and COMBINED this is the (ASCAT) / SMOS / AMSR2 period (from 2012-07-01 onward) for v201912, resp. the (ASCAT) / SMOS /AMSR2 / SMAP period (from 2015-03-31 onward) for v202012.

(3) For ACTIVE this is from 1991-08-05 onward and for PASSIVE and COMBINED after 1978-11-01.

Figure 12 compares the performance of COMBINED v202012 and v201912 to ISMN SM for different land cover types. Only small differences are found here, which can also be related to updates in the ISMN data record (different numbers of stations used). The same applies to Figure 13. However, as expected for both versions, the product performs worse for areas with vegetation coverage than for points that are classified as "Cropland" or "Grassland".

Figure 12: Correlation (Pearson's) for C3S v201912 (left) and C3S v202012 (right) with ISMN (0 – 5 cm sensor depth) for the COMBINED product split by land cover classes. Comparison is for the last merging period of each product (i.e. after 2012-07-01 for v201912, resp. after 2015-03-31 for v202012). The boxes show the median value and interquartile range.

Figure 13: ubRMSD for C3S v201912 (left) and C3S v202012 (right) with ISMN (0 – 5 cm sensor depth) for the COMBINED product split by land cover classes. Comparison is for the last merging period of each product (i.e. after 2012-07-01 for v201912 and after 2015-03-31 for v202012). The boxes show the median value and interquartile range.

2.2. Accuracy – Comparison against Land Surface Models and Reanalyses

2.2.1. GLDAS v2.1

C3S v202012 has been compared against GLDAS v2.1 top layer Soil Moisture (from 2000-01-01 to 2020-12-31 - the end of the C3S v2012012 CDR product). The spatial distribution of the correlation coefficient (Pearson's) and the ubRMSD are shown in Figure 14 and Figure 16 respectively. Note that the GLDAS data are masked in this case for snow covered conditions. Figure 14 shows that the correlation between GLDAS v2.1 and the COMBINED product shows expected spatial patterns. There is a high positive correlation between the products in most temperate zones, with low and near-negative correlations being most prevalent in the northern, boreal regions. The deserts show little to no positive or even negative correlation.

Figure 14: Correlation (Pearson's) of the C3S v202012 COMBINED product with GLDAS v2.1 (covering the time period from 2000-01-01) and comparison with the previous C3S version (right).

Figure 15 shows that the ubRMSD between GLDAS v2.1 and the COMBINED product is low in most regions, with a significant portion being below the 0.1 m³/m³ threshold required in the KPIs (see Section 4). Again, there appears to be a lower agreement between the products in the boreal regions, as it is apparent also in the correlation map.

Figure 15: ubRMSD of the C3S v202012 COMBINED product with GLDAS v2.1 as the reference (left) and comparsion to v201912 (right) - covering the time period from 2000-01-01. The threshold for the KPI (0.1 m³ / m³) is shown in pink (left). Comparison with the previous C3S version (right)- note that ubRMSD is given in the GLDAS value range in the box plots (due to scaling of C3S to GLDAS), i.e- multiplied by 100.

To compare the results for the current and previous version spatially (v202012 vs. v201912), a comparison of the results for correlation (Figure 16) and the ubRMSD (Figure 17) with GLDAS has been undertaken. The impact of SMAP is explored in particular by comparing results of a separate validation run, where only values after March 2015 are considered in all data sets.

Improvements in correlation and reduction in ubRMSD are shown in blue in Figure 16 and Figure 17. In general, the new version shows higher correlations with GLDAS than v201912. However, in some areas (northern Canada, Sahara) a slight degradation is visible when comparing all values after 2000. As noted for the evaluation of previous versions of C3S SM, mainly areas with low data coverage are affected.

When looking at the period that is affected by the addition of SMAP, a significant increase in R (and decrease in ubRMSD) indicates that the quality of the product has overall increased (right-hand panels in Figure 16 and 17).

Figure 16: Difference in Pearson's R for C3S v202012 and v201912, with GLDAS v2.1 as the reference. Blue represents an improvement in R compared to the previous version. Analysis period: 2000-01-01 to 2020-12-31 (left) and 2015-04-01 to 2020-12-31 (right).

Figure 17: Difference in ubRMSD for C3S v202012 and v201912 with GLDAS v2.1 as the reference. Blue represents an improvement in ubRMSD compared to the previous version. Analysis period: 2000-01-01 to 2020-12-31 (left) and 2015-04-01 to 2020-12-31 (right).

Further analysis against GLDAS has been undertaken in the study for the ACTIVE and PASSIVE products in Annex A.

2.2.2. ERA5

The C3S SM v202012 product has been compared to ERA5. ERA5 covers the period from 1979-01-01 to within 3 months of present day and provides soil moisture data for different depth layers. Here, "swvl1" is used, which represents the first 7 cm of the soil. The correlation (Pearson's R, Figure 18) and ubRMSD (Figure 19) between C3S v202012 and ERA5 are presented. The boxplots show intercomparison results to the previous version of C3S SM.

The correlation map (Figure 18, left) shows expected patterns, which are similar to those shown for GLDAS v2.1 (Figure 14, left), the exception being the larger area in North East Siberia, Alaska and North West Canada and parts of Sahara, where in some locations correlations coefficients close to zero or even negative ones are found. Apart from that, high correlations are found in most regions, resulting in a global median of around 0.5, which is higher than the median correlation coefficient with GLDAS.

Figure 18: Correlation (Pearson's) of the C3S v202012 COMBINED product with ERA5 (left, covering the time period after 1979-01-01). Comparison with v201912 COMBINED (right).

The ubRMSD with ERA5 as the reference is higher than the ubRMSD with respect to GLDAS in some areas, with the KPI threshold of 0.1 m³ / m³ being exceeded in some northern areas and some areas of high topographic complexity. This indicates that both satellite and reanalysis/model SM is difficult to estimate for these areas, which are often affected by snow cover and (perma)frost. The global median ubRMSD is 0.055 m³/m³. The IQR of all observations is still below the threshold of 0.1 m³ / m³ (shown in the box plot of Figure 19).

Figure 19: ubRMSD of the C3S v202012 COMBINED product with ERA5 (left), covering the time period from 1979-01-01 onward and comparison with the previous version (right). Areas where the KPI threshold is exceeded are highlighted in pink.

To assess the differences between the current (v202012) and previous version (v201912) of the dataset spatially, the results of the comparison against ERA5 for both products are compared for the correlation coefficient (Figure 20) and ubRMSD (Figure 21). As for the comparison to GLDAS, the period including SMAP is evaluated separately and as expected shows larger differences than the evaluation for the 1979-2020 period.

As for the comparison to GLDAS, a general improvement is seen in v202012 of C3S SM compared to the previous version (v201912). The inclusion of SMAP has a positive effect in terms of R and ubRMSD when C3S SM is compared to ERA5, as seen when products are validated against GLDAS.

Figure 20: Difference in Pearson's R for C3S v202012 and v201912, with ERA5 as the reference. Blue represents an improvement in R compared to the previous version. Analysis period: 1978-01-01 to 2020-12-31 (left) and 2015-04-01 to 2020-12-31 (right).

Figure 21: Difference in ubRMSD for C3S v202012 and v201912, with ERA5 as the reference. Blue represents an improvement in ubRMSD compared to the previous version. Analysis period: 1978-01-01 to 2020-12-31 (left) and 2015-04-01 to 2020-12-31 (right).

A comparison between C3S v201912 and v202012 with ERA5 (period: 2015-04-01 to 2019-03-31) is also available online in QA4SM¹⁰ and has been published at 10.5281/zenodo.4736913

2.2.3. ERA5 Consistency

It is known that there are discontinuities in the ERA5 product¹¹, which can be clearly seen in the example time series presented in Figure 22 for the lower depths. For example, in the top time series, a break in layer 4 can be clearly seen in 2015. However, it should be noted that there is no discernible break in the time series for the upper most soil layer ("swvl1"). However, this assertion is made solely from visual interpretation of the time series. Further assessment should be undertaken to determine if this assertion holds true throughout the data product.

Figure 22: Example time series for volumetric soil water content in the ERA5 product with different depth layers shown in different colours. Two example locations are shown: (top) a desert site in Algeria and a desert site in Mauritania (bottom).

2.2.4. ERA5-Land

In addition to ERA5, global validation was also performed using the ERA5-Land dataset (in the SMAP period). Results are similar to those obtained from ERA5 (Section 2.2.2). Figure 23 shows the improvement in C3S SM v202012 compared with v201912 in terms of R and ubRMSD for the COMBINED product over April 2015 to December 2020 (the time period that includes SMAP data). Equivalent analysis using GLDAS and ERA5 as validation datasets was discussed above (see Figures 16, 17, 20 and 21 respectively).

Figure 23: Difference in R (left) and ubRMSD (right) between C3S v201912 and v202012, with ERA5-Land as the reference. Blue represents an improvement in the new version. Analysis period: 2015-04-01 to 2020-12-31.

An evaluation for Europe is shown in Figure 24 (for Pearson's correlation between C3S SM and ERA5-Land). This has also been published (10.5281/zenodo.4732330) through QA4SM¹².

Figure 24: Pearson's R between C3S SM and ERA5-Land in the period after 2001-01-01 (left) and comparison with previous version in the same area (right).

2.3. Stability – Trend monitoring

Methods for monitoring the stability of the dataset are still under development. Here, preliminary results are presented to demonstrate the methods developed so far along with a discussion of how they will be developed in the future.

2.3.1. Accuracy evolution

To assess the evolution of the C3S SM dataset quality over time, a preliminary analysis is performed of the evolution of accuracy over 2000 to 2020, using ISMN as reference measurements.

Figure 25 (top) shows the evolution of Pearson's R for different land cover types. It shows that the quality of C3S SM (COMBINED) varies slightly, depending on the land cover. Another important factor to consider in this comparison is the number of ISMN stations available in each year, which is represented by the small number below the boxes. Notably there are more ISMN stations available over time. The product appears to be most stable for "Grasslands", while for "Tree Cover" there is visible variation in the product quality before the introduction of ASCAT (2007). "Cropland" shows a slight decline in R over time and a more stable product in the merging period after 2012 compared to the years before. A drop in R is found for 2019 and 2020.

Similar observations can be made in terms of ubRMSD (Figure 25, bottom). Here, the expected wide spread of error for the "Urban areas" class is also visible. This is probably caused by the spatial resolution of C3S SM, where soil moisture networks close to densely populated areas are less representative of the whole C3S SM cell as satellite SM in these areas can be affected by landcover changes (city growth) and RFI.

The COMBINED product of C3S SM is below the ubRMSD KPI threshold of 0.1 m³/m³ in terms of median and IQR. For landcover classes "Grassland" and "Tree Cover" a few points are found which exceed this threshold.

Figure 25: Accuracy evolution of C3S v202012 COMBINED between 2000 and 2020 in terms of Pearson's R (top) and ubRMSD (bottom); based on land cover classes. Numbers at the bottom indicate the number of ISMN stations used in the comparison.

A similar analysis was performed based on four groups of climate classes (described in Section 2.1.6). Figure 26 shows that the performance is most stable for the "BS" (Arid-Steppe) and "Cf / Df" (Temperate-Without Dry Season/Cold Without Dry Season) classes. Larger variation is found for the "Cs / Ds" (Temperate-Dry Summer/Cold-Dry Summer) and the remaining classes ("Other").

The COMBINED product of C3S SM is below the ubRMSD KPI threshold of 0.1 m³/m³ in terms of median and IQR. For climate classes "C/D" (and their subclasses) single points are found outside of this threshold.

Figure 26: Accuracy evolution of C3S v202012 COMBINED between 2000 and 2020 in terms of Pearson's R and ubRMSD; based on climate classes. Numbers at the bottom indicate the number of ISMN stations used in the comparison.

The presented accuracy evolution results are the baseline for the estimation of stability trends in the dataset. Theil-Sen slopes were fitted to the median annual ubRMSD estimates. The so found slope is supposedly representative of the overall stability of the product. Note that this approach is currently under development and might change for future validation activities. Figure 27 shows the distribution of trends for difference classes. At the tested locations for all landcover and climate classes, the change in accuracy metrics over time is very small, which indicates a stable SM product.

Figure 27: Distribution of trends in ubRMSD in C3S SM v202012 COMBINED for different landcover (top) and climate (bottom) classes, tested against ISMN stations, where at least 3 years of accuracy evolution assessment was possible.

2.4. Spatial and temporal completeness

It is important to consider the spatial and temporal completeness and consistency of the product as these can be a key deciding factor for the users in terms of whether the product is suitable for their application.

The spatial and temporal coverage of the product is presented (shown in Figure 28 and Figure 29) in terms of the number of valid (unflagged) observations available. The figures show that the coverage is better in Europe, South Africa and the contiguous US than in some other parts of the world, as well as the improvement in the availability of data post-2015 as SMAP became available (see Table 1 above for further details of sensor periods). This is expected for the product due to the orbital paths of the satellites resulting in higher coverage in equatorial regions. The reduced coverage in boreal and tropical region is expected due to the high Vegetation Optical Depth (VOD) expected in these areas. In addition, the coverage of the northern most latitudes in snow and ice for long periods also reduces the coverage in these areas.

Figure 28: Fractional coverage of the C3S SM v202012 COMBINED product for the ASCAT / SMAP / SMOS / AMSR2 period (2015-04-01 to 2020-12-31). Expressed as the total number of daily observations per time period divided by the number of days spanning that time period.

Figure 29: Fraction of days per month with valid (i.e. unflagged) observations of SM for each latitude and time period for the v202012 COMBINED product. Additional observations from SMAP increase the fractional coverage after 2015.

2.5. Time series analysis

Analysis of time series from a small number of locations provides an insight into the behaviour of the product for different climate and land cover types. Five points have been chosen for which ISMN in situ data are available (and were used in the above assessment). Details of the points are provided in Table 5 and are shown on a global map in Figure 30.

Table 5: Details of locations chosen for time series analysis.

Note: all are classified as having 'medium' soil texture. 

Figure 30: Locations of the points used in the time series comparison (points are given at the C3S GPI location).

The time series (temporally aggregated per month) for the individual locations for the ACTIVE, PASSIVE and COMBINED products are given in Figure 31. In general, the time series appear to follow expected seasonal cycles at each location, i.e. winters are wetter and summers drier and, in the case of GPI 890047 (which is located in Alaska), there are gaps in the data where the location is covered by snow each winter. For previous version of C3S SM, a drop in PASSIVE was observed for most locations around 2011 due to the insufficient intercalibration of AMSR-E and AMSR2. This has been improved for most locations in CDR v202012. Yet for some locations (e.g. GPI 733335 in Figure 31) a sudden drop in SM in still visible. This could be further improved by adding a bridging dataset to C3S SM. FengYun-3B (Yang et al. 2011) is added in ESA CCI SM v6 and could improve the intercalibration in future versions.

Figure 31: Time series of soil moisture (COMBINED/PASSIVE in [m³/m³], ACTIVE in [% sat. / 100])  comparison for the COMBINED, ACTIVE and PASSIVE products of C3S v201912 for the GPIs and land cover types stated for each plot (compared with Figure 30). Note: here, the ACTIVE product is divided by 100 to allow it to be plotted on the same axis as the other products

2.6. Uncertainty analysis

The algorithm used to develop the C3S soil moisture product utilises triple collocation analysis to generate weightings for the combination of different soil moisture observations (Gruber et al. 2017). The SNR calculated as part of the triple collocation process is used to weight the sensors within that merging period. In combination with error propagation techniques, a per-pixel uncertainty is provided within the C3S soil moisture product in the “sm_uncertainty” field.

The weights used within the SMAP / SMOS / AMSR2 period (PASSIVE product) are shown in Figure 32. These weightings show that AMSR2 performs best in very dry areas, such as the Sahara desert, while SMAP is often used in areas with denser vegetation. SMOS overall has the lowest weight in the latest merging period of the PASSIVE product.

Figure 32: Weightings used for merging SMAP (red) / SMOS (blue) / AMSR2 (green) within the C3S v202012 product. These maps are spatially gap filled by regression analysis between the signal-to-noise ratio used in the generation of the weights and vegetation optical depth (VOD).

Figure 33 shows the actual SNR values for the new SMAP sensor. In line with Figure 32 it can be seen that SMAP performs well for sparsely vegetated areas (and often better than other passive sensors for areas with dense vegetation), while the performance over e.g. Eastern Europe, North Canada or South-East-China is lower. In those areas SMOS is then assigned a higher merging weight.

Figure 33: Signal to noise ratio from Triple Collocation analysis for SMAP, after SNR-VOD gap filling is applied. SNR is given on a logarithmic scale (in [dB]). Positive values indicate a higher signal power than noise power.

The evolution of the “sm_uncertainty” field per latitude over the duration of the C3S v202012 product is shown in Figure 34.

It is expected that the uncertainty associated with the product reduces over time. This indicates that in sensor periods closer to the present, the original products are much closer together in absolute values than in the sensor periods nearer to the start of the C3S product. Throughout the product time period, the uncertainties are always higher at latitudes where there is higher vegetation cover, for example at 10 degrees south. This is expected as soil moisture is harder to retrieve in these areas, and there is higher variance in the product where this is the case. It can be seen that the addition of SMAP in 2015 reduced the overall uncertainty in the COMBINED product significantly.

Figure 34: Monthly averages of the uncertainty variable associated with the C3S SM v202012 COMBINED product per latitude over time

3. Application(s) specific assessments

3.1. European State of the Climate 2020

The C3S v202012 PASSIVE data are used in the "European State of the Climate 2020" report produced by ECMWF¹³. In the report, the C3S SM v202012 PASSIVE SM anomaly data (Figure 35, right) are compared against ERA5 SM anomalies (Figure 35, left). The anomalies shown match well with ERA5, although satellite SM seems to be wetter than reanalysis SM, especially in the northern parts of Europe. However, anomalies may be affected by differences in the availability of data under frozen soil conditions (ERA5 is gap-free).

Figure 35: Seasonal soil moisture anomaly for 2020 in ERA5 (left) and C3S v202012 PASSIVE (right). From the "European State of the Climate 2020" report.

3.2. South American Drought

South America experienced extremely dry conditions towards the end of 2020, which was reported by the media as well as multiple meteorological organisations¹⁴. This drought reached its peak in October 2020, and can be clearly seen in SM for November 2020 in C3S v202012 COMBINED (Figure 36). The Pantanal (located mostly in South-Western Brazil) was experiencing the most severe drought in 50 years and severe wild fires were reported across Paraguay at the same time.

Figure 36: Soil Moisture anomalies for November 2020 in the C3S SM v202012 COMBINED product (available via https://dataviewer.geo.tuwien.ac.at/).

4. Compliance with user requirements

The requirements for the C3S soil moisture product are a set of KPIs defined from consideration of user and GCOS requirements. The KPIs are shown in Table 6.

Table 6: Key Performance Indicators (KPIs) for the C3S Soil Moisture Product

An independent accuracy assessment has been conducted comparing the product against in situ observations with different conditions taken into account, i.e. soil depth, land cover etc. In general, there is a variability in the correlation between the datasets, with correlations ranging from 0.4 to above 0.8, depending on the conditions and the locations of the in situ stations used. The ubRMSD, which can be directly taken as a measure of accuracy, is demonstrated to be below 0.10 m³ / m³ for all the different conditions analysed, including land cover type (currently cropland, grassland and tree cover are considered). Therefore, the minimum threshold KPI for the C3S product for accuracy has been met globally. However, the GCOS target requirement of 0.04 m³/m³ is only met under certain land cover (grassland) and climate (semi-arid) conditions. Notice, however, that the use of in situ data for satellite validation is complicated by the presence of representativeness errors, which inflates the actual errors. The effect of representativeness will be addressed in future quality assurance activities, as a finer categorisation of the product skill according to land cover type or vegetation cover.

A comparison against the LSMs GLDAS v2.1, ERA5 and ERA5-Land has also been undertaken to provide a wider global view of the product. The comparison against GLDAS v2.1 has shown expected results, with the KPI threshold of 0.1 m³ / m³ being met in almost all regions (the exception being a few areas with high topographic complexity). However, for the ERA5 and ERA5-Land comparison there are several areas (especially in the North) where the KPI threshold target is not met. Further investigation is needed to ascertain the source of these discrepancies; it is possible that these are a result of insufficient masking in the SM product itself or due to low data coverage in combination with high uncertainties which are generally present in satellite SM retrievals in subarctic areas.

The stability of the C3S product has been assessed in terms of the change in yearly accuracy compared to ISMN stations for all years after 2000. The accuracy between the products (ubRMSD) has been calculated per year and the trends in the accuracy were also analysed. The KPI threshold for stability of 0.05 m³ / m³ / y is met when assessed using this method for all tested stations. More methods to assess SM stability are currently under investigations and will be presented in future evaluation studies.

5. Annex A: Outcomes of the ACTIVE and PASSIVE quality control

5.1. Introduction

The ACTIVE and PASSIVE products of C3S SM v202012 were also compared to the reference datasets described in Section 2. As expected, the COMBINED product outperforms the PASSIVE and ACTIVE product in most locations as shown in Figure 37 for the comparison to GLDAS v2.1.

Figure 37: Intercomparison of the COMBINED, ACTIVE, PASSIVE product of C3S v202012, with GLDAS v2.1 as the reference – plots shown are for Pearson's R (left) and ubRMSD (right). Only common locations and timestamps in all intercompared products are considered.

Figure 38 shows the same comparison for the SMAP period and with ERA5 as the reference. Also here the comparison shows – as expected - a higher correspondence for COMBINED than for ACTIVE or PASSIVE.

Figure 38: Intercomparison of the COMBINED, ACTIVE, PASSIVE product of C3S v202012, with ERA5 as the reference – plots shown are for Pearson's R (left) and ubRMSD (right) for observations after 2015-04. Only common locations and timestamps in all intercompared products are considered.

5.2. Increase in number of observations in ACTIVE

Analysis of data coverage of the ACTIVE product revealed that there is an increase in number of observations in 2020 (see Figure 39, right). This increase was also found in ICDR v201812 and v201706 as well as in ESA CCI SM v4, v5 and v6. Therefore it was concluded that this is not an issue in the C3S processing scheme, but an anomaly in the original input data (caused by presumably a shift in the orbit of MetOp-A). In CCI/C3S, measurements from ASCAT-A and B are temporally resampled and averaged for each day. In 2020 the observation time changed in some cases, which lead to differences in time stamps of the temporally resampled values (while before there were more days where ASCAT-A and B observations were averaged, in 2020 there are more days where only A or B is available). Hence this affects the number of observations in C3S ACTIVE, but not the actual values (as measurements from ASCAT A and B are almost identical), and therefore also not the quality of the data. However, this will be further investigated in ESA CCI SM.

Figure 39: Number of valid observations in the ACTIVE product of v201912 (left) and v202012 (right).

Figure 39 (left) also shows an unexpected drop in "n_obs" in 2018 for v201912 and that this was resolved in the new version by reprocessing the ASCAT input records. CDR v202012 now shows the expected homogeneous coverage from 2015 to 2020, with the expected increase in 2015 due to the addition of ASCAT-B observations.

5.3. Resolved: Strong negative trends in the PASSIVE product in AMSRE/AMSR-2 period

Quality control of the previous C3S version (v201912) revealed that in some locations the PASSIVE product shows a negative break (see Figure 40, left), which is not apparent in the COMBINED or ACTIVE products: there was a strong drop in SM after the introduction of AMSR2 in 2012, caused by insufficient intercalibration of AMSR-E and AMSR2. This issue has been resolved in the latest version of C3S SM by scaling AMSR2 to AMSR-E (using the first, respectively last 3 years of each sensor as there is no overlap between them). Figure 40 (right) shows that the issue is resolved in the new version.

Figure 40: Hovmoeller diagram of ESA CCI SM v4.7 PASSIVE SM anomalies (same algorithm used in C3S v201912, left), which shows the drop in SM after 2011 (climatology period 1991-2010). ESA CCI SM v5.2 (same algorithm used in C3S v202012, right), which shows the improved intercalibration.

This negative break also affected SM anomalies calculated from the PASSIVE product. Before CDR v202012, the PASSIVE SM over Europe was too dry, as shown in Figure 41. Anomalies from v202012 are now more similar to those from ERA5.

Figure 41: Satellite SM anomalies (climatology period 1991-2010) for the PASSIVE C3S SM product from v201912 ICDR (left) and v202012 CDR (middle). The improved intercalibration leads to less extreme SM anomalies, which are now more similar to ERA5 anomalies (right, in [%]).

5.4. Differences in long-term SM trends between the three products

Long term trends in SM data sets are key to understanding the impact of climate change on global soil moisture. With more than 40 years of data, the assessment of long term trends in C3S SM is possible. The COMBINED product has been used for analyses like this before (Dorigo et al. 2012). Ideally, one would expect similar trends in the ACTIVE, COMBINED and PASSIVE product. However, as Figure 42 shows there are differences and trends are often contradictory. This occurs especially in high latitudes (most likely related to data availability) but also in other areas.

Figure 42: Differences in long term (Theil-Sen median) trends. For ACTIVE (top left) the time span 1991-2020 was used, for PASSIVE / COMBINED (top right / bottom) 1978-2020. Note: different scales are shown on these maps such that the colours shown in each case are similar to one another; this is a result of the different absolute values provided in each data product.

The current suggestion is to use the COMBINED product for long-term studies. However, different trends indicate insufficient intercalibration of some sensors or artificial trends within single products, as is demonstrated in Section 5.5.

5.5. Wetting trends in the ASCAT data

An artificial wetting trend is found in ASCAT SSM. This affects the ACTIVE product (and the COMBINED product to a lesser extent) of C3S SM, as well as derived anomalies. Figure 43 shows anomalies over Europe for the year 2019 as in ESA CCI SM v4 dataset (same algorithm as C3S v201912, left column). Note that the ACTIVE product has not significantly changed between v201912 and v202012, hence the issues described here still apply to the latest version. The right column shows the same anomalies when using an experimental version of ASCAT SSM with trend correction in the backscatter dry and wet reference in ESA CCI SM. Artificial trends in the radar observed backscatter signal are probably due to landcover changes and/or RFI (Ticconi et al. 2017). Cities, in particular, stand out in the SM anomaly maps of the uncorrected ACTIVE product. The issue was first detected over Europe, but also affects other regions worldwide.

It is also noted that the issue seems most prominent in spring, and therefore may be (partly) related to the vegetation correction used in the ASCAT product.

Figure 43: Change in ESA CCI SM v4 annual anomalies (for the year 2019) due to backscatter trend correction in ASCAT SSM. The climatological period for all plots is from 1991-2010. Note the difference scales for each of the three products.

The trend corrected product of ASCAT SSM will be used in ESA CCI SM and C3S SM, if officially released and provided by H SAF.

6. Annex B: Detailed comparison of C3S v202012 against C3S v201912

6.1. Introduction

This Annex provides a detailed comparison of the newest dataset version (v202012) against the previous version (v201912). The aim is to determine that the dataset has been made to specification. Differences between the products, which may be of interest to users when using the data in their own applications, are highlighted.

6.2. Comparison of data coverage

The number of valid observations available in each product version has been compared for the ACTIVE, PASSIVE and COMBINED products. Figure 44 shows changes in number of valid observations for each month, averaged by latitude.

Figure 44: Change in (relative) number of valid observations between v201912 and v202012 in COMBINED (top), ACTIVE (bottom left) and PASSIVE (bottom right). Green indicates an increase compared to the previous version, red a decrease in observations.

Due to the use of the new LPRM version (v6 in C3S SM v202012; instead of v5 in v201912) the data coverage increased in many locations especially for the earlier periods. The impact of SMAP after 2015 is also clearly visible especially in the PASSIVE product. Yet there are some locations where the data coverage decreased (in 2010 across all latitudes) and around 15° N between 1998 and 2003. This is in line with data coverage of ESA CCI SM v5 (and the change in data coverage between ESA CCI SM v4 and v5) and is due to changes in the input data (new LPRM) as well as the updated CDF matching algorithm. This is also found in the spatial comparison of "n_obs" in Figure 45. While the input data records for the PASSIVE product have changed significantly in this version, the same input data stream for ACTIVE was used. Yet some differences are found in the data coverage, especially in 2018 (compare Figure 44 bottom left). This is due to missing observations in the ACTIVE product of v201912 (compare the drop in observations in Figure 39), which are now filled (also indicated in Figure 45, bottom left)

Figure 45: Increase in number of unflagged observations in C3S SM v202012 compared to v201912 in the period after 2000-01-01 for the COMBINED (top), ACTIVE and PASSIVE (bottom) products. Only days where GLDAS Noah Soil Temperature > 4° C are considered here. Transition from green to blue for ACTIVE at 366 observations (maximum expected temporal extension versus filling of previously missing values).

6.3. Comparison of time series

The locations, which are compared in Figure 46 for the different product versions, are the same as in Figure 30 in Section 2.5. Overall, the products appear to be similar at these locations. For most points the expected increase in data coverage is found; also the temporal extension of the product is clearly visible. GPI 733335 shows some outlier values at the beginning of 1983 and 2018, which is probably related to the new LRPM version and/or the additional sensor in this version. For other locations (e.g. 756697) these extreme values were reduced in the new version.

Figure 46: Time series of soil moisture (in [m³/m³]) for the different land cover classes considered (GPI locations shown in Figure 30). Showing the data for the COMBINED product from v201912 and v202012 aggregated to 10-daily time steps.

6.4. Comparison of daily images

Daily images for 2019-07-01 for each of the ACTIVE, PASSIVE and COMBINED products have been compared for C3S v201912 and v202012 (difference between them). Figure 47 shows that large differences are found in the PASSIVE product (due to the new sensor), some smaller changes are found for COMBINED for the same reason. ACTIVE shows almost no changes, except for two locations in the North-East (possibly due to the reprocessed ASCAT data). Overall the changes in SM are relatively small considering the value range for SM (0-1 for COMBINED and PASSIVE and 0-100 for ACTIVE).

Figure 47: Difference between C3S SM v202012 and v201912 for the daily COMBINED product on 2019-07-01.

6.5. Comparison of global statistics

To demonstrate the differences between the previous C3S version (v201912) and the current dataset (v202012), the global statistics have been computed for each dataset version and are provided in Table 7. These are for the latest merging period for each dataset.

The statistics show very little difference between the data product versions and indicate that the version are overall similar.

Table 7: Dataset statistics for the different C3S versions CDRs for the latest merging period for each product. The numbers given are the mean values across all GPIs in the dataset, i.e. the mean of the time series for one GPI is calculated and then the mean is taken from all GPIs.

7. Annex C – Validation of Monthly and Dekadal Data

In addition to the daily C3S SM files, monthly and 10-daily (dekadal) averaged files are provided. These include the number of averaged observations per file, the average SM (same unit as the daily data) and fields for the sensors and frequency bands that are merged.

Evaluation of these data was carried out with ISMN v20210131 as the reference. The original ISMN observations were filtered first (only 'G'-flagged measurements are kept) and then averaged to 10-daily and monthly time series.

Figure 48 shows the comparison of "n_obs" between separate validation runs of C3S v201912 and v202012 COMBINED to ISMN (monthly and dekadal). The increase in number of observations (mainly due to the temporal extension of the data record) is visible.

Figure 48: Number of observations that were used in the validation of aggregated C3S SM to (aggregated) ISMN SM for the monthly (left) and dekadal (right) products.

Notably, there is currently no threshold for a minimum number of observations per location in the daily images that are average. The representativeness of averaged images for the actual 10-daily and monthly SM conditions depends on the data coverage for each aggregation period. This should be considered - especially for early periods of C3S SM and for areas with low data density - when using the aggregated files.

Figure 49 shows a comparison of C3S SM v201912 and v202012 for COMBINED monthly and dekadal products. Their performance is similar.

Overall – as expected – the averaged products perform slightly better than the daily products in terms of R and ubRMSD. Therefore the monthly and dekadal data is also below the KPI threshold for the ubRMSD of 0.1 m³/m³.

Figure 49: Intercomparison of COMBINED v201912 and v202012 (monthly, first row) and dekadal (second row) data in terms of Pearson's R (left column) and ubRMSD (right column) with ISMN as the reference.

References

(JCGM), J.C.F.G.I.M. (2008). International vocabulary of metrology — Basic and general concepts and associated terms (VIM). VIM3: International Vocabulary of Metrology, 3, 104

Albergel, C., Dorigo, W., Reichle, R.H., Balsamo, G., de Rosnay, P., Muñoz-Sabater, J., Isaksen, L., de Jeu, R., & Wagner, W. (2013). Skill and Global Trend Analysis of Soil Moisture from Reanalyses and Microwave Remote Sensing. Journal of Hydrometeorology, 14, 1259-1277
BSI (2015). BSI Standards Publication BS EN ISO 9000:2015 Quality management systems Fundamentals and vocabulary. In

de Jeu, R., van der Schalie, R., Paulik, C., Dorigo, W., Pasik, A., Scanlon, T., Preimesberger, W., Kidd, R., & Reimer, C. (2021). C3S Algorithm Theoretical Basis Document (ATBD): Soil Moisture (v202012). In

Dorigo, W., de Jeu, R., Chung, D., Parinussa, R., Liu, Y., Wagner, W., & Fernández-Prieto, D. (2012). Evaluating global trends (1988–2010) in harmonized multi-satellite surface soil moisture, 39

Dorigo, W., Scanlon, T., Buttinger, P., Pasik, A., Preimesberger, W., Paulik, C., & Kidd, R. (2021a). C3S Product User Guide (PUG) and Specification (v202012). In

Dorigo, W., Scanlon, T., Preimesberger, W., Kidd, R., & Chatzikyriakou, C. (2020). C3S SM Product Quality Assessment Report (PQAR) D2.SM.2-v2.0 In. https://datastore.copernicus-climate.eu/documents/satellite-soil-moisture/C3S_312b_Lot4.D2.SM.2-v2.0_202004_Product_Quality_Assessment_Report_v1.0.pdf

Dorigo, W., Scanlon, T., Preimesberger, W., Pasik, A., Kidd, R., Chatzikyriakou, C., & Ng, T. (2021b). C3S Product Quality Assurance Document (PQAD): Soil Moisture (v202012). In

Dorigo, W., Van Oevelen, P., Wagner, W., Drusch, M., Mecklenburg, S., Robock, A., & Jackson, T. (2011). A new international network for in situ soil moisture data. Eos, 92, 141-142

Dorigo, W., Wagner, W., Albergel, C., Albrecht, F., Balsamo, G., Brocca, L., Chung, D., Ertl, M., Forkel, M., Gruber, A., Haas, E., Hamer, P.D., Hirschi, M., Ikonen, J., de Jeu, R., Kidd, R., Lahoz, W., Liu, Y.Y., Miralles, D., Mistelbauer, T., Nicolai-Shaw, N., Parinussa, R., Pratola, C., Reimer, C., van der Schalie, R., Seneviratne, S.I., Smolander, T., & Lecomte, P. (2017a). ESA CCI Soil Moisture for improved Earth system understanding: State-of-the art and future directions. Remote Sensing of Environment

Dorigo, W.A., & Gruber, A.a. (2015). Evaluation of the ESA CCI soil moisture product using ground-based observations. Remote Sensing of Environment, 162, 380--395

Dorigo, W.A., Scanlon, T., & Chung, D. (2017b). C3S Product Quality Assurance Document (PQAD): Soil Moisture. In

Dorigo, W.A., Xaver, A., Vreugdenhil, M., Gruber, A., Hegyiová, A., Sanchis-Dufau, A.D., Zamojski, D., Cordes, C., Wagner, W., & Drusch, M. (2013). Global Automated Quality Control of In Situ Soil Moisture Data from the International Soil Moisture Network. Vadose Zone Journal, 12, 0

Entekhabi, D., Reichle, R.H., Koster, R.D., & Crow, W.T. (2010). Performance Metrics for Soil Moisture Retrievals and Application Requirements. Journal of Hydrometeorology, 11, 832-840

Fang, L., Hain, C.R., Zhan, X., & Anderson, M.C. (2016). An inter-comparison of soil moisture data products from satellite remote sensing and a land surface model. International Journal of Applied Earth Observation and Geoinformation, 48, 37--50

Gruber, A., Dorigo, W.A., Crow, W., Wagner, W., & Member, S. (2017). Triple Collocation-Based Merging of Satellite Soil Moisture Retrievals, 1-13

Gruber, A., Dorigo, W.A., Zwieback, S., Xaver, A., & Wagn, W. (2013). Characterizing Coarse-Scale Representativeness of in situ Soil Moisture Measurements from the International Soil Moisture Network. Vadose Zone Journal, 12

Liu, Y.Y., Dorigo, W.A., Parinussa, R.M., De Jeu, R.A.M., Wagner, W., McCabe, M.F., Evans, J.P., & Van Dijk, A.I.J.M. (2012). Trend-preserving blending of passive and active microwave soil moisture retrievals. Remote Sensing of Environment, 123, 280-297

Plummer, S., Lecomte, P., & Doherty, M. (2017). The ESA Climate Change Initiative (CCI): A European contribution to the generation of the Global Climate Observing System. Remote Sensing of Environment

Preimesberger, W., Scanlon, T., Su, C.H., Gruber, A., & Dorigo, W. (2021). Homogenization of Structural Breaks in the Global ESA CCI Soil Moisture Multisatellite Climate Data Record. IEEE Transactions on Geoscience and Remote Sensing, 59, 17

Ticconi, F., Anderson, C., Figa-Saldana, J., Wilson, J.J.W., & Bauch, H. (2017). Analysis of Radio Frequency Interference in Metop ASCAT Backscatter Measurements. Ieee Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 10, 2360-2371

van der Vliet, M., van der Schalie, R., Rodriguez-Fernandez, N., Colliander, A., de Jeu, R., Preimesberger, W., Scanlon, T., & Dorigo, W. (2020). Reconciling Flagging Strategies for Multi-Sensor Satellite Soil Moisture Climate Data Records, 12, 3439

Wagner, W., Dorigo, W., de Jeu, R., Fernandez, D., Benveniste, J., Haas, E., & Ertl, M. (2012). Fusion of Active and Passive Microwave Observations To Create an Essential Climate Variable Data Record on Soil Moisture. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, I-7, 315--321

WMO (2016). The Global Observing System for Climate (GCOS): Implementation Needs. In (p. 325)

Yang, H., Weng, F., Lv, L., Lu, N., Liu, G., Bai, M., Qian, Q., He, J., & Xu, H. (2011). The FengYun-3 Microwave Radiation Imager On-Orbit Verification. IEEE Transactions on Geoscience and Remote Sensing, 49, 4552-4560

Related articles

• Page:
  SM 2020: Target Requirements and Gap Analysis Document (TRGAD)
• Page:
  SM v202012: Algorithm Theoretical Basis Document (ATBD)
• Page:
  SM v201912: Product Quality Assessment Report (PQAR)
• Page:
  SM v 201912: Algorithm Theoretical Basis Document (ATBD)
• Page:
  SM v201812: Algorithm Theoretical Basis Document (ATBD)