Contributors: E T. Carboni (UKRI-STFC RAL SpaceUsedly (Deutscher Wetterdienst), G.E. Thomas (UKRI-STFC RAL Space)

Issued by: STFC RAL Space (UKRI-STFC) / Elisa CarboniDeutscher Wetterdienst / Tim Usedly

Date: 31/05/2023 14 Jan 2025 

Ref: C3S2_D312a_Lot1.2.1.5-v4.0_2023057_202408_PQAR_CCICloudPropertiesECV_CLD_SLSTR_v1.2

Official reference number service contract: 2021/C3S2_312a_Lot1_DWD/SC1

...

Anchor
history_of_modifications history_of_modifications

History of modifications

Anchor
list_of_datasets list_of_datasets

List of datasets covered by this document

...

Anchor
related_documents related_documents

...

Related documents

Acronyms

...

...

Acronym

...

Definition

...

AATSR

...

Advanced Along-Track Scanning Radiometer

...

AMSR-E

...

Advanced Microwave Scanning Radiometer – EOS

...

ATBD

...

Algorithm Theoretical Basis Document

...

ATSR

...

Along-Track Scanning Radiometer

...

bc-RMSE

...

Bias Corrected Root Mean Squared Error

...

BRDF

...

Bidirectional Reflectance Distribution Function

...

C3S

...

Copernicus Climate Change Service

...

CALIOP

...

Cloud-Aerosol Lidar with Orthogonal Polarization

...

CCI

...

Climate Change Initiative

...

CDR

...

Climate Data Record

...

CER

...

Cloud Effective Radius

...

CFC

...

Cloud Fractional Cover

...

COT

...

Cloud Optical Thickness

...

CTH

...

Cloud Top Height

...

CTP

...

Cloud Top Pressure

...

CTT

...

Cloud Top Temperature

...

CWP

...

Cloud Water Path

...

DARDAR

...

raDAR/liDAR

...

ECV

...

Essential Climate Variable

...

ENVISAT

...

Environmental Satellite

...

ERS

...

European Remote Sensing

...

ESA

...

European Space Agency

...

GCOS

...

Global Climate Observing System

...

ICDR

...

Interim CDR

...

IWP

...

Ice Water Path

...

LWP

...

Liquid Water Path

...

MODIS

...

Moderate Resolution Imaging Spectroradiometer

...

RAL

...

Rutherford Appleton Laboratory

...

SLSTR

...

Sea and Land Surface Temperature Radiometer

...

STFC

...

Science and Technology Facilities Council

...

SYNOP

...

Surface synoptic observations

...

TCDR

...

Thematic Climate Data Record

List of Figures

List of Tables

General definitions

The "CCI product family" Climate Data Record (CDR) consists of two parts. The ATSR2-AATSR Cloud Properties CDR is formed by a TCDR brokered from the ESA Cloud_cci project and an ICDR derived from the SLSTR on board of Sentinel-3. ICDR uses the same processing and infrastructure as the TCDR. Both TCDR and ICDR data have been produced by STFC RAL Space.

These Cloud Properties datasets from polar orbiting satellites consist of these variables: Cloud Fractional Cover (CFC), Cloud Phase (water/ice), Cloud Optical Thickness (COT), Cloud particle Effective Radius (CER), Liquid/Ice Water Path (LWP/IWP), and Cloud Top Pressure (CTP), Height (CTH) and Temperature (CTT).

...

b=\frac{\sum_{i=1}^N (p_i - r_i)}{N} \ \ (Eq. 1)

...

Anchor
acronyms acronyms

Acronyms

Anchor
general_definitions general_definitions

General definitions

B_{i,j}=F_{Data,i,j}-F_{Ref,i,j}

MB=\frac{1}{m*n}*\sum_{i=1}^m\sum_{j=1}^n (w_j(B_{i,j})) 

MAB=\frac{1}{m*n}*\sum_{i=1}^m\sum_{j=1}^n w_j*|B_{i,j}-MB|

Anchor
list_of_figures list_of_figures

List of Figures

Anchor
list_of_tables list_of_tables

List of Tables

Anchor
scope_of_the_document scope_of_the_document

Scope of the document

This document provides a description of the product validation results for the Sea and Land Surface Temperature Radiometer (SLSTR) v4.0 based Interim Climate Data Record (ICDR) of the Essential Climate Variable (ECV) Cloud Properties (CLD).

The dataset produced by RAL Space and Brockmann Consult (BC) under the Copernicus Climate Change Service (C3S) programme ranges from 01/2017 – 12/2023 and provides an Interim Climate Data Record (ICDR) to the brokered Thematic Climate Data Record (TCDR) from European Space Agency Cloud Climate Change Initiative (ESA’s Cloud_cci).

The TCDR is a brokered product based on processing of the (Advanced) Along-Track Scanning Radiometer ((A)TSR) onboard ERS-2 and Envisat by RAL Space for the ESA Cloud_cci programme and ranges from 06/1995 – 04/2012. Detailed validation methodology and results are presented in the Cloud_cci Product Validation and Intercomparison Report [D1].

The ICDR is derived with a five-year gap from SLSTR onboard the Sentinel-3A and -3B satellites spanning from 01/2017 – 12/2023

Anchor
executive_summary executive_summary

Executive summary

The Sea and Land Surface Temperature Radiometer onboard Sentinel-3A has provided data since January 2017. The launch of Sentinel-3B in October 2018 makes it possible to deliver not only individual data from both satellites but also a merged Sentinel-3A/3B product. The merged version (10/2018 - 12/2023) is validated against the following satellite-based datasets: MODIS, CALIPSO-CALIOP and CLARA-A3, as well as ECMWF’s Reanalysis product ERA5. In addition to the merged SLSTR version, a second version on a different grid (equal area in addition to equal angle) is provided for the period from 07/2022 to 12/2023 and also validated against the same reference datasets as the equal angle version of SLSTR.

Validation to these SLSTR derived products is described in the following chapters of this document: Chapter 1 provides a summary of the product validation methodology while chapter 2 presents the validation results. A detailed validation methodology can be found in the Product Quality Assurance Document (PQAD) [D2]. Chapters 3 and 4 discuss possible application specific assessments and compliances with user requirements, respectively. SLSTR shows good agreement to CALIPSO and ERA5 for Cloud Fraction (CFC), while larger differences occur for CLARA-A3 and MODIS. Significant differences occur for Cloud Top Height (CTH), Cloud Top Pressure (CTP) (except for MODIS) and Cloud Top Temperature (CTT). Also, Ice Water Path (IWP) and Liquid Water Path (LWP) show consistently higher values for the SLSTR dataset. Differences between the two provided grid versions from SLSTR are negligible and meet the goal requirement by GCOS.

Calculated biases are evaluated against the GCOS requirements and are summarized in Table 1.

Anchor
chapter1 chapter1

1. Product validation methodology

Detailed information about the validation methodology can be found in the corresponding PQAD [D2], section 3. The validation process is separated into three parts: Data preparation (section 1.1), validation (section 1.2) and evaluation (1.3).

Anchor
section1.1 section1.1

1.1 Data preparation

Table 1-1 provides a summary of the datasets used for the validation and their temporal availability, spatial- and temporal resolution.

All datasets are, if necessary, remapped to 1°x1° spatial resolution by bilinear interpolation.

Anchor
section1.2 section1.2

1.2 Validation

Following uncertainty metrics are calculated: Bias, Mean Bias and Mean Absolute Bias.

Bias is the difference of dataset and reference dataset for each month and grid box:

B_{i,j}=F_{Data,i,j}-F_{Ref,i,j} \ (1)

With B as Bias and F as dataset/reference and i, j as indices. Prior to the bias calculation, the datasets are collocated and only grid point considered, where two (or more) datasets have valid values (not nan). Grid points with identical grid points set to nan for a different dataset are set to nan.

Mean Bias (MB) describes the overall bias with respect to a reference dataset. It is defined as the bias of two gridded data records and a subsequently calculation of the global spatial average. This results in one value per month which can be averaged over the whole time period.

MB=\frac{1}{m*n}*\sum_{i=1}^m\sum_{j=1}^n (w_j(B_{i,j})) \ (2)

with MB as Mean Bias, i and j (m and n, respectively) as indices, w as cosine weighting factor and B as Bias.

Mean Absolute Bias (MAB) is a bias corrected uncertainty metric and calculated by subtracting the previously calculated MB from every grid box bias. Subsequently the same steps as for the calculation of the mean bias are applied.

MAB=\frac{1}{m*n}*\sum_{i=1}^m\sum_{j=1}^n w_j*|B_{i,j}-MB| \ (3)

Anchor
section1.3 section1.3

1.3 Evaluation

The previously calculated Mean Absolute Bias is used as evaluation against the requirements defined by the Global Climate Observing System (GCOS) in The 2022 GCOS ECVs Requirements (GCOS 245) [D3]. They are summarized in Table 1-2.

Anchor
chapter2 chapter2

2. Validation results

Sections 2.1 – 2.6 show the validation results for the six variables CFC, CTP, CTT, CTH, IWP and LWP with a climatology of collocated, deseasonalized, centered and weighted global averages. After the collocation the seasonality of each dataset is removed from the climatology as well as the average of each dataset subtracted. Absolute values show larger differences depending on the variable and the centered climatologies allow to evaluate the stability.

Anchor
section2.1 section2.1

2.1 Cloud Fractional Cover

Image Added

The merged SLSTR product shows overall good agreement with the reference data records in terms of stability (see Figure 2-1) and representation of the annual cycle. Although it has the lowest average of cloud fraction (61%, compared to references with 62%-68%, see Figure 2-2) it shows best agreement to CALIPSO and ERA5 leading to fulfill the goal requirement defined by GCOS. Bias between the two different SLSTR grid versions are negligible (61% cloud fraction on average). Largest differences occur for MODIS with generally higher values on large parts of the globe (see Figure 2-3). Differences in CLARA-A3 are separated over land and ocean areas: CLARA-A3 appears to have higher values over the oceans whereas large parts of land (North/South America, Greenland, Europe, Asia) have a positive bias. Differences compared to ERA5 are biggest over the Antarctica and north of 60°N. While these biases are negative, 60°N-60°S shows a slight positive bias. Comparison with CALIPSO shows a similar pattern with negative biases from 60° poleward. Other than CLARA-A3, differences over the oceans are positive, while land areas show a negative anomaly.

Image Added

Image AddedImage Added

Image AddedImage Added

Image Added

Anchor
section2.2 section2.2

2.2 Cloud Top Pressure

Image Added

Cloud Top Pressure (as well as Cloud Top Height (see section 2.4) and Cloud Top Temperature (see section 2.3)) shows significant differences in comparison with CALIPSO and CLARA-A3 (Figure 2-6). The SLSTR versions on equal angle and equal area grid (653.17 hPa and 655.63 hPa) have a comparable average to MODIS (667.88 hPa) but much higher values compared to CALIPSO (439.80 hPa) and CLARA-A3 (502.88 hPa) (Figure 2-5). Figure 2-4 shows that the stability is overall good compared to MODIS and CALIPSO. CLARA-A3 shows a significant negative trend from early 2023 on which is due to increased satellite drift.

Image Added

Image AddedImage Added

Image AddedImage Added

CALIPSO as well as CLARA have positive biases all over the globe except for the eastern half of the Antarctica. Biases are highest in tropical regions indicating that SLSTR underrepresents high clouds (cirrus) which results in higher cloud top pressures.

Anchor
section2.3 section2.3

2.3 Cloud Top Temperature

Image Added

Results for Cloud Top Temperature are similar to Cloud Top Pressure, with significant higher values for SLSTR (Figure 2-7, Figure 2-9). This is in accordance with Cloud Top Pressure/Height. CLARA-A3 should be treated with caution as a reference due to drifting afternoon orbits (NOAA-18/19) (Figure 2-7). Differences occur primarily in the topics (Figure 2-8). There are no differences between the equal area and equal angle grid version.

Image AddedImage Added

Image Added

Anchor
setion2.4 setion2.4

2.4 Cloud Top Height

Image Added

Cloud Top Height is significantly higher for CALIPSO (8.03 km) and CLARA-A3 (6.71 km) (Figure 2-11) confirming the results for CTP and CTH. Differences between the two SLSTR grid versions are small (4.24 km and 4.21 km for equal angle and equal area, respectively). Figure 2-10 shows an overall good stability between SLSTR and CALIPSO and a similar negative trend as CLARA-A3 shows. In addition, a slight positive trend from early 2023 on is seen for all datasets. Figure 2-12 confirms the findings with negative anomalies in tropical regions and positive anomalies in over the Antarctica.

Image Added

Image AddedImage Added

Image Added

Anchor
section2.5 section2.5

2.5 Ice Water Path

Image Added

Comparison of Ice Water Path and Liquid Water Path is limited to 50°N-50°S. While reference datasets MODIS and ERA5 show relatively good agreement to each other (also in terms of stability, see Figure 2-13). SLSTR has not just higher values (0.19 kg/m², compared to 0.08 kg/m² for CLARA-A3, 0.02 kg/m² for ERA5 and 0.05 kg/m² for MODIS) (Figure 2-14) but also a positive trend for the entire period of 10/2018 – 12/2023. Positive biases are seen all over the globe with highest values around the Inter Tropical Convergence Zone (Figure 2-15).

Image Added

Image AddedImage Added

Image AddedImage Added

Anchor
section2.6 section2.6

2.6 Liquid Water Path

Image Added

Validation of Liquid Water Path shows in general similar patterns compared to Ice Water Path: Reference datasets CLARA-A3 (average 0.08 kg/m²), ERA5 (0.06 kg/m²) and MODIS (0.03 kg/m²) (Figure 2-17) show good agreement to each other, while SLSTR (0.12 kg/m²) has higher values with a stronger annual cycle and trend (Figure 2-16). Positive bias is clearest visible over ocean areas, while there is no or partly a small negative bias over land area (e.g. parts of Africa and North America in comparison with CLARA-A3). Compared to CLARA-A3, one can see a negative bias in mountainous regions for IWP and LWP (e.g. Rocky Mountains, Andes, Alps and parts of the Himalayan region) (Figure 2-18).

Image Added

Image AddedImage Added

Image AddedImage Added

Anchor
chapter3 chapter3

3. Application(s) specific assessments

This section is not applicable. There are no additional application specific assessments known since the dataset has just been published.

Anchor
chapter4 chapter4

4. Compliance with user requirements

The GCOS requirements [D3] for the ECV Cloud Properties are used to evaluate the compliance for different users needs. Tables 4-1 and 4-2 show the requirements as well as the results.

GCOS defines three requirements depending on user’s needs:

• Goal (G): The strictest requirement, indicating no further improvements necessary
• Breakthrough (B): Intermediate level between threshold and goal. Breakthrough indicates that it is recommended for certain climate monitoring activities
• Threshold (T): Minimum requirement

The SLSTR ICDR meets the breakthrough/target requirement for the horizontal/temporal resolution, respectively.

The accuracy for CFC (depending on the reference between -1.47% and -6.82%) meets the threshold requirement and partly the goal and breakthrough requirement (Table 4-1).

However, there are larger differences for CTH, CTP and CTT which do not meet the requirement (Table 4-2). Despite having higher values, SLSTR LWP and IWP meet the breakthrough (LWP) and threshold (IWP) requirements (Table 4-3).

It is worth mentioning, that the GCOS requirements, defined by the World Meteorological Organisation (WMO), are not focused on satellite-based data records but also on climate models. Satellite-based data records, especially historical observing systems, are often not able to achieve the requirements.

...

bc- RMSE=\sqrt{\frac{\sum_{i=1}^N ((p-b)-r)^2}{N}} \ \ (Eq. 2)

...

Stability: The variation of the bias over a multi-annual time period

Table 1: Summary of variables and definitions

...

Variables

...

Abbreviation

...

Definition

...

Cloud mask / Cloud fraction

...

CMA/
CFC

...

A binary cloud mask per pixel (L2) and from there derived monthly total cloud fractional coverage (L3C)

...

Cloud optical thickness

...

COT

...

The line integral of the absorption extinction coefficient (at 0.55μm wavelength) along the vertical in cloudy pixels.

...

Cloud effective radius

...

CER

...

The area-weighted radius of the cloud droplet and crystal particles, respectively.

...

Cloud top pressure/
height/
temperature

...

CTP/
CTH/
CTT

...

The air pressure [hPa] /height [m] /temperature [K] of the uppermost cloud layer that could be identified by the retrieval system.

...

Cloud liquid water path/
Ice water path

...

LWP/
IWP

...

The vertical integrated liquid/ice water content of existing cloud layers; derived from CER and COT. LWP and IWP together represent the cloud water path (CWP)

Table 2: Definition of processing levels

...

Processing level

...

Definition

...

Level-1b

...

The full-resolution geolocated radiometric measurements (for each view and each channel), rebinned onto a regular spatial grid.

...

Level-2 (L2)

...

Retrieved cloud variables at full input data resolution, thus with the same resolution and location as the sensor measurements (Level-1b).

...

Level-3C (L3C)

...

Cloud properties of Level-2 orbits of one single sensor combined (averaged) on a global spatial grid. Both daily and monthly products provided through C3S are Level-3C.

Table 3: Definition of various technical terms used in the document

Scope of the document

This document provides a description of the product validation results for the Essential Climate Variable (ECV) Cloud Properties. This CDR comprises inputs from two sources: (i) brokered products from the Cloud Climate Change Initiative (ESA's Cloud_cci), namely those coming from processing of the Advanced Along-Track Scanning Radiometer (A)ATSR) data and (ii) those produced under this contract for the Climate Data Store, specifically those coming from processing of the Sea and Land Surface Temperature Radiometers (SLSTR).

The Thematic Climate Data Record (TCDR) is the product brokered from the European Space Agency Cloud Climate Change Initiative (ESA Cloud_cci) ATSR2-AATSR version 3.0 (Level-3C) dataset. This is produced by RAL from the second Along-Track Scanning Radiometer (ATSR-2) on board the second European Remote Sensing Satellite (ERS-2) spanning the period 1995-2003 and the Advanced ATSR (AATSR) on board ENVISAT, spanning the period 2002-2012.

In addition, the Interim Climate Data Record (ICDR) is the product derived from the Sea and Land Surface Temperature Radiometer (SLSTR) on board of Sentinel-3 and spans the period from January 2017 to present. Validation for this SLSTR derived product for the period from January 2017 to June 2022 is described in this document. In addition to the validation of the individual products from Sentinel-3A and -3B, the merged product (Sentinel-3A+3B) is also validated spanning the period from October 2018 to present (Sentinel-3B provides data from October 2018 on).

Executive summary

This document provides a description of the product validation results for some of the Essential Climate Variable (ECV) Cloud Properties. These specific products are brokered to (in case of (A)ATSR) or produced for the Climate Data Store (in the case of SLSTR) by the Copernicus Climate Change Service (C3S).

The TCDR is a brokered version of ESA's Cloud_cci ATSR2-AATSR version 3.0 (Level-3C) dataset, produced by RAL from the second Along-Track Scanning Radiometer (ATSR-2) on board the second European Remote Sensing Satellite (ERS-2) which operated in the period 1995-2003 and the Advanced ATSR (AATSR) on board ENVISAT which operated in the period 2002-2012. In addition, the Sea and Land Surface Temperature Radiometer (SLSTR) on board of Sentinel-3 has been operating from 2017 to present and provides the input to the ICDR. The validation of the ICDR is over the period from January 2017 to June 2022 with not just the individual products from Sentinel-3A/B but also the merged product spanning the period from October 2018 to June 2022. The retrieval algorithm is presented in [D2] and the validation methodology refers to the Cloud_cci Product Validation and Intercomparison Report [D1]. The same methodology is applied to the SLSTR dataset.

Poulsen et al. (2019) [D3] is the paper describing the dataset that includes cloud properties as well as Surface Radiation Budget and Earth Radiation Budget products. This document will mainly refer to the Cloud_cci Product Validation and Intercomparison Report [D1].

Table 2-1 provides a summary of the estimated accuracies of the TCDR together with the GCOS requirements. The validation results are provided in [D1] section 7-2, with a recommendation for use.

For the ICDR, the intercomparison with MODIS (using only the first 5 years of SLSTR data) show biases consistent with values found for the TCDR vs MODIS comparison (D1, section 4.1) for Cloud Fractional Cover (CFC), Cloud Top Pressure (CTP) and Liquid Water Path (LWP), but higher value for Ice Water Path (IWP) SLSTR-B, with a global averaged bias of -25 g/m³ for SLSTR-A and -40 g/m³ for SLSTR-B (TCDR IWP bias with MODIS was -29 g/m³).

This document is divided in different sections: 

• the first section presents a brief description of validation methodology together with a series of references for further information;
• the second section present the results of the validation and comparison of TCDR and ICRD data.
• the third section presents the compliance with user requirements and include recommendation on the usage and know limitations

1. Product validation methodology

The validation methodology is described in section 2.4 of [D1]. In addition, the validation methodology is also described in the corresponding Product Quality Assurance Document (PQAD) [D4]. In summary, the methodology uses the bias between the Cloud_cci product and the reference data to estimate the accuracy of the dataset. The bias corrected root mean squared error (bc-RMSE) is used to express the precision of CDR compared to a reference data record, which is also known as the standard deviation about the mean. Stability is calculated as the variation of the bias over a multi-annual time period.

...

2.1 Validation results for the TCDR

The validation results for TCDR products are provided in [D1], section 3 and 4. The evaluation is divided in validation against high quality and satellite-based reference observations Please find information on the sensors in [D1]: CALIOP (Annex A.1), AMSR-E (A.2), DARDAR (A.3) and MODIS-C6.1 (A.10) (CALIOP, AMSR-E and DARDAR) and an intercomparison to well-established, satellite-based cloud datasets of similar kind (e.g. MODIS Collection 6.1). Table 2-1 provides a summary of the resulting TCDR accuracies.

...

Product name

...

GCOS targets

...

Cloud CCI dataset

...

Comments

...

Cloud Fractional Cover (CFC)

...

Accuracy

...

5 %

...

-5.1 %

...

Level-2 validation against CALIOP

...

Stability (per decade)

...

3 %

...

-0.52 %

...

Level – 3C (L3C) comparisons to MODIS C6.1

...

Cloud Top Height (CTH)/ Pressure (CTP)

...

Accuracy (low/mid/high)

...

0.5/0.7/ 1.6 km

...

0.12 km (liquid cloud)
-1.76 km (ice cloud)

...

Level-2 validation against CALIOP

...

Stability (per decade)

...

15 hPa

...

0.45 hPa

...

Level-3C (L3C) comparisons to MODIS C6.1

...

Cloud Optical Thickness (COT)

...

Accuracy

...

10 %

...

No validation possible due to a lack of reliable reference data. Used to estimate LWP and IWP, that are validated.
Values here present the global bias in comparison with MODIS Level-3

...

Stability (per decade)

...

2 %

...

-0.03 % (liquid cloud)

...

L3C comparisons to MODIS C6.1

...

Liquid Water Path (LWP)

...

Accuracy

...

25 %

...

-2.4%

...

Level-2 validation against AMSR-E

...

Stability (per decade)

...

5 %

...

-0.06 %¹

...

L3C comparisons to MODIS C6.1

...

Ice Water Path (IWP)

...

Accuracy

...

25 %

...

-39.9 %

...

Level-2 validation against DARDAR

...

Stability (per decade)

...

5 %

...

-0.04 %²

...

L3C comparisons to MODIS C6.1

...

Cloud Effective Radius (CER)

...

Accuracy

...

10 %

...

No validation possible due to a lack of reliable reference data. Variable is based on LWP and IWP.
Values here present the global bias in comparison with MODIS Level-3

...

Stability (per decade)

...

1μm

...

-0.96 μm (liquid)
-0.33 μm (ice)

...

L3C comparisons to MODIS C6.1

1 Value obtained from the difference between CDR LWP trend (0.99g/m²/decade) and MODIS LWP trend (8.06g/m²/decade) divided by mean MODIS C6.1 LWP (123g/m²).
2 Value obtained from the difference between CDR IWP trend (-2.27g/m²/decade) and MODIS IWP trend (7.82g/m²/decade) divided by mean MODIS C6.1 IWP (208g/m²).

2.1.1 Cloud Fractional Cover (CFC)

Cloud Fractional Cover (CFC) is validated against CALIOP and reported in [D1], section 3.1.1 and compared with MODIS and reported in [D1], section 4.1.1. A slight underestimation of cloud occurrences is found in the Cloud_cci data compared to CALIOP, which is primarily due to a lack of sensitivity of passive imager data with respect to optically very thin clouds.

2.1.2 Cloud Top Height (CTH)

Cloud Top Height is validated against CALIOP and presented in [D1] section 3.1.3.

For liquid clouds only very small biases (0.12km) and bc-RMSE (0.97km) are found.

For ice clouds, the strong underestimation of cloud top height is evident and it is a common feature for all three Cloud_cci datasets. It is mainly caused by high-level, optically thin clouds. Biases are around -3.5 km and bc-RMSE around 2.3 km. Removing the optically very thin cloud layers at the top of the CALIOP profiles, improves the agreement between Cloud_cci and CALIOP substantially.

2.1.3 Cloud Top Pressure (CTP), Cloud Optical Thickness (COT) and Cloud Effective Radius (CER)

These parameters are compared against MODIS. As the quantities are also derived from satellite observations, it is not valid to calculate measures of bias and RMSE, as the MODIS values cannot be necessarily considered to be better. Nonetheless a comparison is useful. This comparison is documented in the following sections of [D1]: Cloud Top Pressure (CTP) is compared in section 4.1.2. Cloud Effective Radius (CER) is compared in section 4.1.5 and 4.1.6. Cloud Optical Thickness (COT) is compared in section 4.1.3 and 4.1.4 for liquid and ice cloud.

2.1.4 Liquid Water Path (LWP) and Ice Water Path (IWP)

Validation of Liquid Water Path (LWP) is presented in [D1] section 3.1.4. It is carried out against AMSR-E products and compared with MODIS and reported in [D1] section 4.1.7.

Ice Water Path (IWP) is validated against the DARDAR IWP product and reported in [D1] section 3.1.5 and compared with MODIS which is in section 4.1.8.

Validating liquid water path over ocean against AMSR-E gives very convincing results for the CDR dataset, with bc-RMSE values of 25 g/m², only small biases (-1.44 g/m²) and high correlations (0.76). Validating ice water path against the combined CALIPSO-CloudSat product DARDAR shows good agreement with correlations of 0.45. There is a general underestimation of IWP by Cloud_cci which in terms of relative bias partly exceeds 50%.

2.2 ICDR comparison with MODIS

The first 5.5 years and 3.5 years (of SLSTR-A and SLSTR-B respectively) of products have been compared against MODIS (Collection 6.1 Terra) following the same methodology described in [D1] section 4.1. We estimate the bias, i.e. mean differences, and the monthly mean global average of C3S and the MODIS data. To compute the monthly mean global average of both datasets we considered only the valid data between -60° and +60° latitude.

Table 2-2 shows the bias results from this comparison for both TCDR data (from D1) and for the ICDR. Except for the IWP, all the properties show better results (lower bias) for the ICDR in comparison to TCDR.

Figure 2-1 and Figure 2-2 show an example of ICDR monthly products for March 2017 and the equivalent monthly product from MODIS. These figures are for illustrative purposes so the user knows what to expected. Nonetheless, note that for this month, the ICDR and MODIS datasets are spatially similar across all four properties. However, there are some small differences observed. For example CTP seems much higher in the tropics in the MODIS product compared to the ICDR. For a more detailed analysis please go to the [D1].

...

...

3. Application(s) specific assessments

In addition to the extensive product validation (see chapter 2 for results and chapter 2/3 in [D4] for validation methodology) a second assessment is introduced to evaluate the Interim Climate Data Record (ICDR) against the Thematic Climate Data Record (TCDR) in terms of consistency. Since frequent ICDR deliveries make detailed validation not feasible, a consistency check against the deeply validated TCDR is used as an indication of quality. This is done by a comparison of the following two evaluations:

• TCDR against a stable, long-term and independent reference dataset
• ICDR against the same stable, long-term and independent reference dataset

The evaluation method is generated to detect differences in the ICDR performance in a quantitative, binary way with so called Key Performance Indicators. The general method is outlined in [D6] chapter 3. The same difference between TCDR/ICDR and the reference dataset would lead to the conclusion that TCDR and ICDR have the same quality (key performance is "good"). Variations or trends in the differences (TCDR/ICDR against reference) would require a further investigation to analyze the reasons. The key performance would be marked as "bad". The binary decision whether the key performance is good or bad is made in a statistical way by a hypotheses test (binomial test). Based on the TCDR/reference comparison (global means, monthly or daily means) a range is defined with 95% of the differences are within. This range (2.5 and 97.5 percentile) is used for the ICDR/reference comparison to check whether the values are in or out of the range. The results could be the following:

• All or a sufficient high number of ICDR/reference differences lies within the range defined by the TCDR/reference comparison: Key performance of the ICDR is "good"
• A smaller number of ICDR/reference differences is within the pre-defined range: Key performance of the ICDR is "bad"

3.1 Results

The results of the KPI test are summarized in Table 3-1.

...

p2.5

p97.5

...

-1%

3%

...

-7.9 hPa

6.93 hPa

...

-16.4 g/m²

11.3 g/m²

...

-4.65 g/m²

4.48 g/m²

...

Sentinel-3B:

...

Percentiles were calculated based on the comparison of the TCDR using the Advanced Along Track Scanning Radiometer (AATSR) instrument against MODIS Collection 6.1 (Terra satellite only) as reference dataset for the variables Cloud Fractional Cover (CFC), Cloud Top Pressure (CTP), Ice Water Path (IWP) and Liquid Water Path (LWP). Percentiles were based on the time from 2002-2012 with monthly means and applied to the ICDR from 01/2017 (10/2018) to 06/2022 for Sentinel-3A (Sentinel-3B and merged product Sentinel-3A+B) based on measurements of the Sea and Land Surface Temperature Radiometer (SLSTR).

Most of the ICDR months are outside the TCDR-based KPI limits and leading to “bad” KPI tests. Therefore, the ICDR is not stable in relation to the TCDR. This is due to multiple reasons starting with the fact of a five year gap (2012-2016) between TCDR and ICDR. In addition, TCDR and ICDR are based on different instruments with SLSTR on Sentinel-3 and (A)ATSR/ATSR-2 on Envisat/ERS-2, respectively. Differences occur due to a lower bias between ICDR and reference dataset and a subtraction of the monthly means (based on the TCDR) to remove the annual cycle leads to values outside of the KPI range (see method in [D6], chapter 3.2.2). On the other hand, IWP has a higher bias compared with MODIS which can be caused by different cloud classification due to differences in the visible channels between (A)ATSR and SLSTR. Please note that significant changes between 01/2017 - 12/2020 and 01/2017 - 12/2021 are due to bugfixes.

4. Compliance with user requirements

The validation results for the TCDR products are presented and described in detail in [D1], section 7. In this section, a summary highlighting recommendations on usage is presented. More detailed information about the user requirements is provided in the Target Requirements and Gap Analysis Document (TRGAD) [D5].

Table 2-1 (section 2) provides an overview of the GCOS requirements for the Cloud Properties and the values achieved by TCDR. It should be noted that GCOS requirements are targets and are often not attainable using existing or historical observing systems. The Cloud_cci doesn't meet the frequency requirement (3h) due to the nature of the satellite observations, but exceeds the spatial resolution (50 km GCOS target). ICDR accuracies (estimated with the first 5 years of data) are consistent with TCDR accuracies apart from IWP where we find slightly higher bias in comparison with MODIS (section 2.2).

For nearly all validations for which a trusted reference data source is available, the compliance to GCOS requirements could be shown, e.g. cloud fractional cover stability, cloud top height accuracy and stability, liquid water path accuracy and stability and ice water path stability. Cloud fractional cover accuracy is close to the GCOS target requirements. For effective radius and optical thickness no reliable reference data is available for accuracy compliance analysis.

A general problem is the assessment of the stability. Stability assessments are based on comparisons to MODIS, which in turn however, is not an entirely reliable source itself as it is sometimes characterized by significant trends which may or may not be true.

The following are recommendations on the usage and known limitations (from [D1] table 7.1):

Cloud Fractional Cover (CFC)

• Discrimination of heavy aerosol and cloud is not optimal, thus aerosol is sometimes flagged as clouds in such conditions. It is advised to be careful in the interpretation of cloudiness in periods with dust / volcanic ash outbreaks. Cloudiness is overestimated in these conditions.
• Cloud detection during polar night over snow and ice is generally difficult.
• Cloud detection in twilight conditions is of rather poor quality due to the small number of channels used.
• The ATSR2-AATSR cloud detection consistency between ATSR2 and AATSR is not optimal.
• Due to a limitation of passive imagers, the cloud fraction is usually biased towards lower values, compared to CALIOP, for example.

...

• In semi-transparent (ice) cloud conditions, the cloud top will be assigned too low.
• Multi-layer clouds are not modelled hence the CTH for cases of an upper layer of thin cirrus will effectively retrieve a radiative height (approx. 1 optical depth into the cloud).

...

• COT is a daytime product only.
• In cases the wrong phase (liquid/ice) is assigned, the optical thickness is likely to have significant errors.
• In the case of incorrectly assigned surface BRDF the optical depth is likely to be biased. With high BRDF the COT will be biased low. With low BRDF the COT will be biased high.
• In case of sub-pixel clouds or cloud borders the COT is likely to have significant errors.
• In case of optically thin clouds above (especially poorly) a defined highly reflecting surface, the COT retrieval might be problematic.
• For very optically thick clouds, the measurements go into saturation and thus the sensitivity of the measurement to the COT is small. Those values should be accompanied by large uncertainty values.

...

• CER is a daytime product only
• In cases where the wrong phase (liquid/ice) is assigned, the effective radius is likely to have significant errors.
• In case of sub-pixel clouds or cloud borders, the effective radius is likely to have significant errors.

...

• Monthly mean LWP is computed using only daytime measurements.
• Since LWP is computed from retrieved COT and CER, the same limitations as for COT and CER apply for LWP.
• The method used assumes vertically homogeneous clouds, which might deviate from the true situation. In case of vertically inhomogeneous cloud layers, e.g. multi-layered clouds, the LWP retrieval is likely to show large errors, since the CER is retrieved from the uppermost cloud layers and may not be representative for the entire vertical column.
• In cases of a wrongly assigned cloud phase, i.e. ice cloud is treated as liquid cloud, the retrieved LWP will show large errors.

...

...