Last modified on Jan 12, 2024 11:29

Contributors: Karl-Göran Karlsson (SMHI), Amelie Mayer (DWD), Benjamin Würzler (DWD), Tim Usedly (DWD), Jan Fokke Meirink (KNMI), Gareth Thomas (UKRI-STFC RAL Space), Elisa Carboni (UKRI-STFC RAL Space)

Issued by: SMHI/Karl-Göran Karlsson

Date: 06/10/2023

Ref: C3S2_D312a_Lot1.3.1.1-2022_TRGAD-CLD_v1.1

Official reference number service contract: 2021/C3S2_312a_Lot1_DWD/SC1

History of modifications

Related documents

Acronyms

List of tables

List of figures

General definitions

Climate data records

Climate data compilations from observations are most often referred to as Climate Data Records (CDRs). However, the data records from satellites may consist of different types of quantities, from original radiances to derived products. Radiance data of climate quality are defined as Fundamental Climate Data records (FCDRs) while data records consisting of satellite-derived geophysical products are defined as Thematic Climate Data Records (TCDRs). In the ideal case the TCDRs should be derived by methods using FCDRs as input. However, if standards for the used radiances have not fulfilled the strict requirements for being classified as FCDRs, these radiances may be denoted Fundamental Data Records (FDRs). Notice that TCDRs can currently be based on either FCDRs or FDRs.

A special case of TCDRs are data records produced with short latency (e.g., shortly after the end of a month). These are called Interim Climate Data Records (ICDRs). The word Interim means that the data record has a higher uncertainty than the original TCDR since it has not been possible to use exactly the same input data as for the TCDR due to the short latency. Interim also means that a user may have to wait for the next edition of the TCDR to get a fully consistent and homogenous climate data record that includes data from the period with ICDR data. Normally ICDRs behave very similar to TCDRs but continuous monitoring of their quality is recommended.

Notice that since ICDRs are continuous extensions of the TCDR they are also delivered at subsequent times in separate batches (numbered 1,2,3..etc) where each one covers a certain time period (e.g. a number of months). Thus, when formally describing the full ICDR in the text (i.e., using the name specified in the delivery list), the ICDR version number is given but the batch number is written in generic form using letter x, for example ICDR v1.x. This is just to indicate that the batch number is only describing a temporal increment of the product and not any change of the product.

Uncertainty parameters

The meaning of the terms uncertainty, accuracy and error is often difficult to interpret and may be treated differently in various referred documents. In this document we adopt the following interpretation:

The accuracy, uncertainty or error of an estimated ECV (or, more formally, Thematic Climate Data Record, TCDR) is described by three differently contributing components:

1. The systematic error

2. The random error

3. The time-dependent error

The systematic error is commonly the mean error or the Bias. For non-Gaussian distributions of the error the median or the mean absolute error can be a more useful quantity.

The random error is commonly the root-mean-squared deviation RMSD. Sometimes the Bias is subtracted yielding the centered root-mean-squared deviation cRMSD. Notice that if the Bias is zero the two mentioned quantities are equal and may be interpreted as the standard deviation of the error (often denoted standard error).

The time-dependent error is commonly the change in Bias over time (for ECVs or TCDRs over decades). We call this parameter stability.

All TCDRs are normally evaluated against target requirements for the systematic, random and time-dependent error.

Testing the quality and consistency of TCDRs and ICDRs

This C3S project also deals with extensions of TCDRs, i.e. products derived from continued processing of the CDRs using the same methods and algorithms as originally used for TCDR production. We denote these CDRs Intermediate Climate Data Records, ICDRs. To evaluate the ICDR compliance with original TCDRs, a different approach in terms of defined requirements is followed. The ICDR is assessed on the basis of the TCDR distribution with respect to a reference validation source. After calculation of this distribution of differences, the ICDR is evaluated against the same reference and a binomial test is applied to verify that 95 % of the difference values for the ICDR lie within the upper and lower bounds of the TCDR difference distribution. The lower and upper bounds of the difference distribution is defined as the 2.5^th and 97.5^th percentiles of the difference distribution.

For further clarity, a binomial test is a way to test the statistical significance of deviations by referring to a theoretically expected distribution of observations. In this case, we use the theoretically expected distribution of observation differences which is estimated from the difference between TCDR results and corresponding results from a validation source. We now want to test if a corresponding but restricted, i.e., based on a shorter time series of ICDR results, difference distribution is similar in its shape to the original TCDR difference distribution. This can be tested by selecting one upper and one lower percentile in the original distribution (here, the 2.5^th and 97.5^th percentiles) and check how many samples will fall within or outside this restricted distribution if randomly extracting a number of samples. The resulting distribution of yes and no answers as a function of the number of samples can be described by the binomial distribution (see statistical standard literature for its definition). Consequently, this sample-based difference distribution from the ICDR can then be numerically compared with what could be expected from the reference distribution based on the TCDR. Based on this, one can judge whether the ICDR results are representative or not for the TCDR results. Deviations here would then indicate particular problems for the ICDR products (assuming that the character of reference observations does not change).

More details on the estimation of errors and uncertainty parameters are given in the Report on Updated KPIs (D10).

Product requirements

Depending on the data record producer, different product requirements may be applied and they are used to evaluate validation results. An often-used way to handle this is to define several levels of requirements where each level is linked to specific needs or priorities. A three-level approach like the following is rather common:

Satellite product levels

Satellite-based products are often described as belonging to the following condensed description of processing levels, each one with different complexity and information content:

A more comprehensive definition of all processing levels is given here: https://www.earthdata.nasa.gov/engage/open-data-services-and-software/data-information-policy/data-levels.

Radiation terms

Since satellite measurements are primarily about radiation measurements in different parts of the spectrum, some definitions or synonyms need to be explained. Roughly, the spectrum is usually sub-divided into one part where solar radiation dominates and one part where radiation emitted by the Earth and the atmosphere dominates.

The solar part is usually referred to as “visible (VIS)” radiation and covers approximately wavelengths smaller than 1 µm. Two sub-regions are often referred to, namely “ultraviolet (UV)” for radiation below approximately 0.38 µm, and “near-infrared (NIR)” for radiation between 0.78 µm and 1 µm (but sometimes claimed to continue up to 2.8 µm).

The part dominated by emitted radiation from the Earth is often referred to as “thermal” radiation. Common synonyms used are “infrared (IR)” or “terrestrial” radiation. Also here, we have several sub-regions defined. The “short-wave infrared (SWIR)” region is approximately defined by wavelengths between 1 µm and 2.5 µm. The “medium-wave infrared (MWIR)” region is approximately defined by wavelengths between 2.5 µm and 5 µm. The “long-wave” region (often simply referred to as just “infrared” to represent the bulk majority of radiation emitted by the Earth) defines radiation from approximately 5 µm up to about 1 mm. Radiation above 1 mm up to 10 cm is denoted “microwave (MW)” radiation.

Special terms

The term “AVHRR-heritage” is frequently used in the document. By this is meant spectral channels of other sensors than the AVHRR which show a close similarity (or heritage) to the AVHRR channels, i.e., having almost the same spectral characteristics.

A product is said to be “brokered” when an existing data record from an external source (i.e., not produced exclusively within this C3S project) is handled. This also means that target requirements for these products are set to their achieved validation results since the product was not developed and validated in the C3S project.

We can get a better idea of how accurate the final product values are by using the method of “error propagation”. It means that the retrieval method is capable of accounting for errors or uncertainties in the measurements or products used to derive the final product, e.g., radiances, input or ancillary data. In this way, the uncertainty of the final products can be estimated.

Radiation fluxes are sometimes described as being “balanced”. It comes from the fact that instrument uncertainties for radiation budget measurements are often too high to be capable of providing accurate estimations of the net radiation fluxes at the top of atmosphere. Thus, balancing is a form of bias correction based on investigations of energy balance from other observations and model studies.

Calibration of radiances are sometimes described as based on “vicarious” methods. This indicates that there is no on-board mechanism on the satellite that provides the necessary calibration information. Consequently, parameters used in calibration equations have to be estimated retrospectively from historic data by use of additional references (e.g., for visible radiances, often Earth surfaces which are considered to be invariant or stable are used as reference targets).

An “OPeNDAP” server is an advanced software solution for remote data retrieval (see https://www.opendap.org/).

“Triple Collocation (TC)” is a large-scale validation technique by which error variances and data-truth correlation coefficients of three independent datasets can be estimated without a specific reference observation. For further details, see Stoffelen (1998).

Scope of the document

This document provides relevant information on requirements and gaps for the following two sets of products:

1. The Cloud Properties TCDR AVHRR CLARA v2.0 product (known as CLARA-A2.1) and its associated ICDR v2.x,

2. The Cloud PropertiesTCDR ESA AATSR v3.0 product and its associated SLSTR ICDR v3.1.1 and v4.0.

The document is divided into three parts. Part 1 describes the products the present document refers to. Part 2 provides the target requirements for the products. Part 3 provides a past, present, and future gap analysis for the products and covers both gaps in the data availability and scientific gaps that could be addressed by further research activities (outside C3S).

Executive summary

The products associated with the Cloud Properties TCDR AVHRR CLARA v2.0 (covering the period 1982-2018) are described together with their target requirements. For the four cloud products (i) cloud fractional cover (CFC), (ii) cloud top pressure (CTP), (iii) liquid water path (LWP) and (iv) ice water path (IWP) the mean error requirements are 5 %, 50 hPa, 10 gm^-2 and 20 gm^-2, respectively. Corresponding stability requirements are 2 % per decade, 20 hPa per decade, 3 gm^-2 per decade and 6 gm^-2 per decade, respectively.

Key Performance Indices (KPIs) for ICDR products to be produced for the extension of the TCDR (so far covering the period January 2019 until June 2022) have been defined using MODIS Level 3 products as reference observations. The KPI test is based on a binomial test against low (2.5%) and high (97.5%) percentiles of the MODIS-CLARA difference distribution. The low and high percentiles are given in the following table:

An extensive description of past, current and future availability of data from the Advanced Very High Resolution Radiometer (AVHRR) is given. In addition, future prospects of utilizing AVHRR-heritage spectral channel data from new imaging sensors on new satellites are described. It is concluded that the AVHRR-based observations series, based on one morning and one afternoon orbit constellation, can be prolonged to reach at least 60 years if adding AVHRR-heritage information. However, for this to become effective, efforts are needed to harmonize and homogenize observations between true AVHRR data and AVHRR-heritage data. This concerns both calibration aspects and spatial resolution aspects. Regarding the development of retrieval methods, it is noted that the access to high quality cloud observations from active sensors (especially CALIPSO-CALIOP data) has played an important role in advancing both retrieval methods and methods for uncertainty estimations in recent years. The access to the high-quality reference data has been especially important for the development of Bayesian and artificial neural network (ANN) based retrievals. A future continuation of active observations from space is judged as crucial for further development of retrieval methods based on AVHRR-heritage data.

The cloud products provided by the Cloud Properties TCDR ESA AATSR v3.0 are described, along with their target requirements. In addition to the Cloud Properties provided by the AVHRR CLARA v2.0 TCDR (CFC, CTP, LWP and IWP), the Cloud_cci TCDR also provides (i)cloud top temperature (CTT), (ii) cloud optical thickness (COT) and (iii) cloud effective radius (CER). The mean error requirements for these variables are 5K, 10% and 10%, respectively, with the corresponding stability requirements being 1K, 2% and 2% per decade.

The data record of the (A)ATSR sensors, running from 1995-2012, and the ICDR from the SLSTR sensor (covering 2017-2022) are discussed. Options for filling the four-year gap between the end of the AATSR data record and the beginning of the SLSTR record are also provided. A key advantage of the (A)ATSR and SLSTR data (compared to AVHRR) is that the data are highly stable, both in terms of satellite orbital parameters and instrument calibration, at the cost of temporal coverage and, in the case of (A)ATSR, instrument swath width. With the introduction of the second operational SLSTR sensor on Sentinel-3B in 2018, it is possible for SLSTR to provide nearly global coverage twice a day by combining data from both platforms. For C3S, such a combined product has been provided as the v4.0 data version.

1. Product description

Cloud products estimated from satellite sensors describe one component of the atmospheric part of the global water cycle where the other parts are Water Vapour and Precipitation products. To get a full picture of the global water cycle, components from the Earth´s surface have to be added (e.g., evaporation, ice and snow accumulation/melting and runoff estimates).

Two cloud data records, both based on visible and infrared imagery, are delivered to C3S. One is based on data from the Advanced Very High Resolution Radiometer (AVHRR). AVHRR-data forms the longest available time series of meteorological imagery with observations starting in 1978. Here, a data record with data from 1982 and onwards is delivered. The second data record is based on data from the Along Track Scanning Radiometer (ATSR) and its successor AATSR (i.e., Advanced ATSR). The used observation series started in 1995 and ended in 2012. An extension of the data record, with similar observations from the Sea and Land Surface Temperature Radiometer (SLSTR) starting in 2017, is also provided.

1.1 Cloud Properties TCDR AVHRR CLARA v2.0 + ICDR v2.x

This section describes cloud products of the CLARA-A data record (CM SAF CLoud, Albedo and Radiation data record – AVHRR-based) and its ICDR extension. This data record is produced by the EUMETSAT Satellite Application Facility on Climate Monitoring (CM SAF) and exists currently in three versions: CLARA-A1, CLARA-A2 and CLARA-A2.1. The reason for the difference in version numbering between the C3S and CM SAF projects is that C3S never included the CLARA-A1 data record. The version described here as version 2.0 is the extended version of the second edition of this dataset, denoted CLARA-A2.1 by the CM SAF. In the remainder of this document we will use the shorter notation CLARA-A2.1 to refer to this data record.

The CLARA-A2.1 CDR is based on Advanced Very High Resolution Radiometer (AVHRR) observations onboard the NOAA and EUMETSAT Metop satellites. Observations are available in reduced spatial resolution globally (approximately 5 km, denoted Global Area Coverage or GAC) and the final CDR is compiled in a regular global grid with 0.25̊ (~25 km) resolution. The time period covered by the TCDR is from January 1982 to December 2018. ICDR production started in January 2019 and products have been delivered covering the period from January 2019 until June 2022. Products were based on an updated AVHRR Fundamental Climate Data Record (FCDR) where corrections for the visible channels originated from a method described by Heidinger et al. (2010).

In addition to being based on AVHRR radiances, the CLARA-A2.1 cloud retrieval algorithms make use of the following auxiliary datasets:

• USGS land cover map (Eidenshink and Faundeen, 1994)

• USGS Digital Elevation Map (USGS, 1996)

• ERA-Interim surface and atmospheric profile temperatures and pressure (Dee et al., 2011)

• ERA-Interim vertically integrated moisture contents (Dee et al., 2011)

• ERA-Interim snow depth and albedo (Dee et al., 2011)

• OSI SAF sea ice concentrations (EUMETSAT OSI SAF, 2015)

• MODIS-based surface emissivity (Wan et al., 2015) and surface albedo (Moody et al., 2008)

• Multi Sensor Reanalysis (MSR) total ozone column climatology (Van der A et al., 2010)

Due to the unavailability of ERA-Interim reanalysis data from 2019 onwards, the ICDR production (starting in January 2019) is thereafter utilizing ECMWF Integrated Forecast System (IFS) analyses as auxiliary Numerical Weather Prediction (NWP) input. The alternative to use ERA-5 instead of ERA-Interim as input is not possible with the required latency. The CLARA-A2.1 cloud TCDR and ICDR cover in total six different cloud products (see Karlsson et al., 2017) but for the C3S_312b_Lot1 and C3S2_312a_Lot1 projects exclusively four products were selected. These are described in the following sub-sections. The basic processing chain and detailed algorithm descriptions are given in reference documents D6-D9.

1.1.1 Cloud Fractional Cover (CFC)

This product is derived directly from results of a cloud detection method providing binary cloud masks as Level-2b products (i.e., daily sampled products on a global 0.05° grid). The daily products are converted to cloud fractional cover on a global 0.25° latitude/longitude grid and final monthly means are calculated from the daily means merging results for all included satellites. The cloud fractional cover is defined as the fraction of cloudy pixels per grid square compared to the total number of analysed pixels in the grid square. CFC is expressed in percent.

This product is calculated using the NWC SAF PPS (Polar Platform System) version 2014 (incl. patch 2) cloud mask algorithm (see https://www.nwcsaf.org/ and http://nwcsaf.smhi.se/ for details on the NWC-SAF project). The algorithm (detailed by Dybbroe et al., 2005) is based on a multi-spectral thresholding technique applied to every pixel of the satellite scene. A detailed description is given in D8.

1.1.2 Cloud Top Pressure (CTP)

The Cloud Top Pressure (CTP) is one of the variables that forms the CLARA-A2.1 Cloud Top Level product suite (although all are derived using the same basic algorithms):

1. Cloud Top Temperature (CTT), expressed in Kelvin;

2. Cloud Top Height (CTH), expressed as altitude over ground topography (m);

3. Cloud Top Pressure (CTP), expressed in pressure co-ordinates (hPa).

The product selected for C3S_312b_Lot1 and C3S2_312a_Lot1 is the Cloud Top Pressure product (CTP). However, all three types of the Cloud Top products are accessible (i.e., all three products are included in the original brokered product files) and they are all mutually compatible.

The CTP product is derived using two approaches, one for opaque and one for fractional and semitransparent clouds, and they are applied to all cloudy pixels as identified by the PPS cloud mask product. The opaque algorithm matches measured 11 μm radiances with cloud free and cloudy TOA 11 μm radiances simulated from reference atmospheric profiles. The semi-transparent algorithm estimates mean CTT from two-dimensional histograms of 11 minus 12 μm brightness temperatures versus 11 μm brightness temperatures by use of a curve-fitting technique. As a final step, CTT is converted to CTP using reference atmospheric profiles.

Both schemes are part of NWC SAF PPS 2014 patch 1. Details can be found in D8.

Level 3 CTP products are basically derived in the same way as for the CFC product, i.e., averaging of daily Level 2b CTP products on a global 0.25° latitude/longitude grid. However, the standard linear averaging is complemented with an alternative logarithmical averaging as a consequence of the non-linear decrease of pressure with geometrical height. The logarithmic averaging gives a CTP product that is more consistent with the geometrical CTH product.

1.1.3 Liquid Water Path (LWP)

The liquid water path (LWP, units kg m^-2) is derived exclusively for daytime conditions and exclusively for cloudy pixels having the liquid phase near the cloud top as deduced from the CLARA-A2.1 cloud phase product. The product is calculated from the estimated cloud optical thickness (τ) and the particle effective radius (r_e) following the concept introduced by Nakajima and King (1990). It is based on the principle that the reflectance of clouds at a (for cloud particles) non-absorbing wavelength in the visible region (e.g., 0.6 or 0.8 μm) is strongly related to τ and has little dependence on r_e, whereas the reflectance of clouds at an absorbing wavelength in the shortwave-infrared region (e.g., 1.6 or 3.7 μm) is strongly dependent on effective radius. Lookup Tables (LUTs) of top-of-atmosphere reflectances at the respective pair of channels as a function of viewing and illumination geometry, τ and r_e are defined from radiative transfer calculations, and used in the retrieval process.

Liquid water path is then computed from the retrieved τ and r_e values by (Stephens, 1978): LWP = 2/3 ρ_l τ r_e, in which ρ_l represents the density of liquid water (1000 kg m^-3).

More details on the retrieval scheme can be found in D9.

Level 3 LWP products are averaged linearly onto the 0.25° latitude/longitude grid from Level 2b products for all cloudy pixels with the liquid phase. Results are also calculated for all-sky conditions, i.e. by including cloud-free and ice-cloud pixels as zeroes in the averaging.

1.1.4 Ice Water Path (IWP)

The ice water path (IWP, units kg m^-2) is derived in exactly the same way as LWP but now for cloudy pixels with the frozen (ice) phase near the cloud top, and using  \( IWP = \frac{2}{3} \rho_j \tau r_e \) in which  \( \rho_j \) represents the density of ice (930 kg m^-3). Also here, Level 3 results are available as both cloudy-sky and all-sky averages. Details on the retrieval scheme can be found in D9.

1.2 Cloud Properties TCDR ESA AATSR v3.0+ ICDR ESA SLSTR v3.1.1+v4.0

This section describes the cloud products of the Cloud Properties TCDR ESA AATSR v3.0 data record and its extension with the same retrieval scheme applied to the Sea and Land Surface Temperature Radiometer (SLSTR), which is produced specifically for C3S. In the following, we will refer to this data record as Cloud_cci v3.0. The short name comes from the fact that these products were originally developed in the ESA project with the name ESA_CLOUD_CCI (where CCI stands for Climate Change Initiative). The Cloud_cci v3.0 record is based on Along-Track Scanning Radiometer 2 (ATSR-2) and Advanced Along-Track Scanning Radiometer (AATSR) observations onboard the ESA 2nd European Research Satellite (ERS-2) and ENVISAT satellites, respectively. Together, the data record provided by these two instruments is often abbreviated to (A)ATSR. The SLSTR instrument, which is the successor to (A)ATSR, is on board the Copernicus Sentinel-3 platform.

The SLSTR instrument was designed as the operational successor to the (A)ATSR instruments, using the same measurement principles and techniques, improving them based on the experience gained with the (A)ATSRs, and continuing the 17-year data record provided by ATSR-2 and AATSR (21 years if ATSR-1 is included). Unfortunately, the development time of the Copernicus Sentinel satellites and the demise of ENVISAT in 2012, broke the continuity of this dataset, with an almost five-year gap between the end of the AATSR record and the availability of SLSTR. Despite this, SLSTR products can be considered an ICDR extension of the (A)ATSR TCDR, for the following reasons:

1. The (A)ATSR and SLSTR instruments were conceived with the goal of creating long-term data records for climate monitoring. Consistency and stability are at the core of their design.

2. The instruments are very similar – SLSTR provides a wider swath, some additional channels, increases the spatial resolution of the shortwave channels and alters the viewing geometry compared to (A)ATSR. But the differences in the instrument and orbital characteristics between AATSR and SLSTR are comparable to those between ATSR-2 and AATSR.

The authors would suggest that addressing the (A)ATSR/SLSTR data gap, by linking the two datasets with a third source of data which overlaps with each should be a priority for Copernicus, and would likely be readily achievable given the necessary support. There are several datasets which provide similar measurements to the (A)ATSR/SLSTR Cloud Properties dataset (including the AVHRR CDR provided through C3S itself) and which overlap both (A)ATSR and SLSTR temporally. Such data could be used to assess the consistency of the (A)ATSR TCDRs and SLSTR ICDRs, but this remains to be done for the Cloud Properties dataset.

Observations are available on a 1x1 km grid, which closely matches the true instrument spatial resolution globally (so called AATSR-multimission L1b format) and the final CDR is compiled in a regular global grid with 0.1° (~10 km) resolution for daily averages and 0.5° resolution for monthly averages. The covered time period of the data record from (A)ATSR ranges from June 1995 to April 2012. Products were based on the third reprocessing of the AATSR-multimission archive, which included vicarious calibration of the shortwave channels over the entire data record to correct for long-term calibration drift (Smith, 2012). SLSTR products from Sentinel-3A begin in January 2017, with Sentinel-3B products beginning in October 2018. At the time of writing SLSTR data from both platforms is available up to the end of June 2022.

The product was produced using the Community Cloud retrieval for Climate (CC4Cl) processing chain, which is based around the Optimal Retrieval of Aerosol and Cloud (ORAC) retrieval scheme, both of which are described in detail in [D13, D14] and by Sus et al. (2018) and McGarragh et al. (2018).

In addition to being based on (A)ATSR radiances, the Cloud_cci CC4Cl processing chain makes use of the following auxiliary datasets:

• USGS Digital Elevation Map (USGS, 1996)

• ERA-Interim surface and atmospheric profile temperatures and pressure (Dee et al., 2011)

• ERA-Interim profiles of moisture content and Ozone concentrations (Dee et al., 2011)

• ERA-Interim snow depth and albedo (Dee et al., 2011)

• National Snow and Ice Data Center Near-real-time Ice and Snow Extent (NISE) sea ice concentration (Brodzik and Stewart, 2016).

• ERA-Interim 10 m u and v wind components (Dee et al., 2011).

• MODIS-based land surface bidirectional reflectance distribution function (BRDF) data (MCD43C1 Collection 6, (Schaaf and Wang, 2015)).

• Land surface emissivity from the Cooperative Institute for Meteorological Satellite Studies (CIMSS) “Baseline Fit” database.

For (A)ATSR data outside the temporal coverage of these datasets (for example, ATSR-2 data prior to the 1999 launch of MODIS products), climatologies based on these data are used. In the case of the SLSTR ICDR, ERA-5 data is used rather than ERA-Interim.

The six cloud parameters provided to C3S from the Cloud_cci v3 dataset are described below:

1.2.1 Cloud Fractional Cover (CFC)

This product is derived directly from results of a cloud detection method providing binary cloud masks as level-2 products (i.e. derived on the 1 km L1b grid). The 1 km products are converted to cloud fractional cover on a global 0.05° latitude/longitude grid. Monthly means are also calculated from the level-2 products, on a 0.5° latitude/longitude grid, merging results for all included satellites. The cloud fractional cover is defined as the fraction of cloudy pixels per grid square compared to the total number of analysed pixels in the grid square.

This product is calculated using the Community Cloud for Climate (CC4Cl) neural network cloud mask (Sus et al., 2017).

1.2.2 Cloud Top Pressure (CTP) and Cloud Top Temperature (CTT)

Cloud_cci provides three values defining the Cloud Top, all of which are different parameters derived from the same retrieval:

1. Cloud Top Temperature (CTT), expressed in Kelvin;

2. Cloud Top Height (CTH), expressed as altitude over ground topography (km);

3. Cloud Top Pressure (CTP), expressed in pressure co-ordinates (hPa).

The products selected for C3S_312b_Lot1 are the Cloud Top Pressure product (CTP) and Cloud Top Temperature (CTT). However, all three types of the Cloud Top product are accessible (i.e., all three products are included in the original brokered product files) and they are all mutually compatible.

As the ORAC retrieval used in the CC4Cl processor used in Cloud_cci is an optimal estimation scheme, the CTP/CTT are derived through a fit of all channel radiances to a physical model of cloud, thus providing simultaneous derivation of the cloud opacity and emissivity in conjunction with its temperature. However, it can be said that the majority of information on the cloud temperature and height is provided by the 11 and 12 μm radiances observed by the satellite. The relationship between CTT and CTP (and CTH) is defined using reanalysis atmospheric profiles. Details can be found in [D14].

Level 3 CTP products are basically derived in the same way as for the CFC product, i.e., averaging of level-2 CTP products on a global 0.05° or 0.25° latitude/longitude grid.

1.2.3 Cloud Effective Radius (CER)

Cloud droplet or ice-crystal effective radius (CER, units μm) is derived during daylight conditions, as it is heavily constrained by reflectance in the shortwave infrared (e.g. 1.6 or 3.7 μm channels of (A)ATSR – for Cloud_cci, the 3.7 μm channel was not used, as this channel was found to be inconsistent between ATSR-2 and AATSR).

The relationship between the observed TOA radiances and the cloud effective radius is strongly impacted by the phase – liquid or ice water – of the observed cloud (as well as the ice crystal habitat in the case of ice cloud). Cloud phase was determined through the application of the Pavolonis cloud typing algorithm (Pavalonis et al., 2005), with additional tests on retrieved CTT. In the case of liquid water, the cloud particle size distribution is modelled as a modified gamma distribution, whereas ice clouds are modelled using the “general habit mixture” ice scattering properties provided by Baum et al. (2010, 2014).

1.2.4 Cloud Optical Thickness (COT)

The cloud optical thickness, or depth, (COT) is also derived during daylight conditions, and is mainly constrained by the shortwave channels provided by (A)ATSR (e.g. 0.67, 0.87 and 1.6 μm). Note that (A)ATSR also provides a 0.55 μm channel, but this was not used as it is often not active on ATSR-2 and it does not provide significant additional information on Cloud Properties in addition to the 0.67 μm channel.

1.2.5 Cloud Water Path (CWP)

The cloud water path (CWP, units kg m^-2) is a derived product, determined from the retrieved COT and CER during the day, and from the cloud effective emissivity (which represents equivalent microphysical information to optical depth and effective radius in a single parameter, which can be used when the information content of the measurements is too low to derive two separate parameters). The calculation of CWP follows the approach of Han et al. (1994):

\( CWP = \frac{4}{3}\frac{\tau_{c}r_{e}\rho}{Q_{e}} (Eq 1) \)

where  \( \tau_{c} \) is COT,  \( r_e \) is CER, ρ is the density of water (1.0 g cm^-3) or ice (0.917 g cm^-3) and Qe is the extinction coefficient of water (2.0 cm^-1) or ice (2.1 cm^-1) at 550 nm. During night retrievals the value of the  \( \tau_c r_e \) product is constrained by the cloud effective emissivity.

The CWP is calculated from level-2 products (i.e. at 1x1 km resolution). When computing the daily and monthly products provided via the CDS, CWP is split into separate averages for Liquid Water Path (LWP) and Ice Water Path (IWP), the sum of which can be considered the total CWP value.

2. User Requirements

This section describes the requirements which have been set to be achieved by the described products. Requirements can be set at different levels (as explained in the section with General definitions) but here we will focus on what is called the Target Requirements. These requirements define the main goals for data producers which have to be fulfilled by their products. Requirements are specified by the use of various accuracy parameters which are also listed in the section with General definitions. Observe that for brokered products the target requirements are set to the achieved validation results since these products are not developed and tested within the C3S project.

Concerning products to be used in climate monitoring, requirements for what should be achievable through Earth Observation systems are generally defined by the World Meteorological Organisation (WMO) Global Climate Observation System (GCOS) expert panel. However, these requirements are generally oriented towards the capability and resolution of climate models with a rather course spatial resolution while many products listed here focus more on the monitoring of local and regional scale conditions. Also, they are often not attainable using existing or historical observing systems. Thus, GCOS requirements are not always identical to the requirements listed here since also other user groups than the climate modelling community have contributed in setting the requirements. However, the relation to GCOS requirements are discussed below for each individual product.

2.1 Cloud Properties TCDR AVHRR CLARA v2.0 + ICDR v2.x

2.1.1 Introduction to user requirements

There are several user groups for cloud climate data records. At least four major groups can be identified:

• National Meteorological/Environmental Services performing continuous (mostly short-term) climate monitoring activities (e.g., issuing of monthly climate bulletins).

• Institutes and commercial services monitoring and planning solar energy activities.

• The climate research community evaluating climate variability and trends from both satellite and surface observations.

• The climate modelling community evaluating climate model performance and cloud effects in climate simulations.

This document mainly focuses on the user requirements for the climate research and climate modelling community, although some relevant aspects from other user groups are also covered.

To understand requirements better we shortly summarize the major points of interest for each cloud product described above, considering that the current spread in climate model predictions is to a large extent explained by differences in the description of clouds, or more clearly the cloud feedback processes:

Cloud Fractional Cover (CFC):

• CFC basically defines areas being mostly cloud-free or mostly cloudy which roughly regulates how much solar radiation will reach the surface and how much of it is reflected back to space. It also roughly determines to what extent the planetary albedo is dominated by cloud reflection or by surface reflection (e.g., an important factor in the Arctic Ocean).

Cloud Top Pressure (CTP):

• Clouds impose a greenhouse effect in the atmosphere in a similar way as the absorbing greenhouse gases. This effect is highest for high clouds with cold cloud tops. Thin high clouds have the largest warming effect, while for low level clouds the cooling by reflection of solar radiation is larger than the infrared warming, leading to a net cooling effect. Monitoring the balance and distribution between low and high clouds is a central issue in climate monitoring.

Cloud Optical Thickness (COT):

• In order to understand a cloud’s impact on radiation processes, it is necessary to describe how a cloud is changing radiation through scattering processes when radiation is passing through the cloud. Thus, the COT parameter can be used to define the extinction of radiation through scattering processes. COT is also closely linked to cloud emissivity which describes a cloud’s ability to emit radiation. Consequently, COT is an important parameter when estimating the greenhouse effect of clouds (as mentioned above for CTP).

Cloud Effective Radius (CER)

• Scattering of solar radiation is also depending on the particle distribution in the cloud. For example, a cloud with small droplets (e.g. Stratocumulus clouds) is ‘brighter’ than a thin Cirrus cloud with large ice particles. Since the refractive indices for water and ice differ in parts of the short-wave infrared spectrum (e.g., between 1 µm and 4 µm) an average (‘effective’) radius for liquid and ice clouds can be derived from short-wave infrared measurements. Thus, CER is an important parameter for the description of cloud effects in climate models. It is also needed for estimating the two CWP components LWP and IWP (see below).

Liquid Water Path (LWP):

• LWP is the bulk estimate of the amount of (vertically integrated) liquid cloud condensate in the atmosphere. The quantity is of major interest for climate model evaluations since it can be used to understand how well models simulate the water cycle (including condensation and precipitation processes). It also implicitly determines how water clouds affect radiation conditions through the relation with COT and CER in Eq. 1.

Ice Water Path (IWP):

• IWP plays the same role as LWP but for the solid (frozen) water phase.

2.1.2 Specification of requirements

To evaluate the quality of a product, certain validation procedures and rules have to be followed. A good overview is given in the introduction section in the Report on Updated KPIs (D10). For the CLARA-A2.1 cloud products, we have used the Bias, the centered root-mean-squared deviation (cRMSD) and stability as defined previously (General definitions) as quality descriptors.

Some exceptions exist. For example, the mean error of the CTP product can sometimes be entirely dominated by a few very large deviations. A better choice here for CTP could therefore be to use the median and the mean absolute error instead of Bias and cRMSD.

In the CM SAF project, requirements for all products were defined in three levels (threshold, target and optimal) reflecting both the maturity of the retrieval and the estimated optimal capability of the measurement or sensor. For C3S_312b_Lot1 and C3S2_312a_Lot1 we will only discuss one of these levels: the target requirement.

An important aspect is whether requirements are defined for Level 2 (instantaneous) products or for Level 3 (time-averaged) products. Validation of the Level 2 product must be considered as the most accurate evaluation method provided that measurement and reference observations can be adequately matched in space and time, and that a statistically significant time series of evaluated measurements can be compiled. Level 3 products are generally defined at a spatially reduced resolution (through spatial averaging) which means that especially the variability as defined by the precision parameter (cRMSD) is reduced. On the other hand, several additional error sources may alter the Bias parameter, such as insufficient spatial (i.e., where measurements are lacking or only intermittently available) and temporal (i.e., missing days or parts of the daily observations) sampling. A very important error source is also that Level 3 products from reference observations will almost never be possible to compile in exactly the same way as the product to be evaluated. All these additional potential error sources must be taken into account when defining target requirements for Level 3 products.

2.1.3 GCOS requirements

The most relevant requirements for the climate research community are specified by United Nations-ratified program the Global Climate Observation System (GCOS-154, see reference D1). Table 2-1 below lists the current GCOS target requirements for a set of Cloud Properties. The relevant entries in this table for the current C3S_312b_Lot1 and C3S2_312a_Lot1 deliverables are Cloud Amount (CA) (matching CFC), CTP (matching CTP) and Cloud Water Path (CWP) (matching both LWP and IWP). These requirements have been determined by assuming a cloud feedback similar to a radiative forcing of about 0.3 Wm^-2, which is roughly 20 % of the current greenhouse gas forcing.

Table 2‑1: GCOS target requirements for six Cloud Properties: Cloud Amount (CA), Cloud Top Pressure (CTP), Cloud Top Temperature (CTT), Cloud Optical Depth (COD), Cloud Water Path (CWP) and Cloud Effective Radius (CRE). From reference D1 (page 29). Observe that GCOS parameter names differ slightly from what otherwise is used in this document (for example, CA=CFC, COD=COT and CRE=CER)

It should be noted that the interval representation of accuracies and stabilities for Cloud Amount (CA) and Cloud Top Pressure (CTP) separates the cloud dataset into opaque clouds (the lower value) and semi-transparent clouds (the higher value). Also, the meaning of the term Accuracy as defined by GCOS in this table is the Bias parameter used in this C3S project (see General definitions).

Some minor changes to the requirements in Table 2-1 have been introduced recently (in 2016) in reference D2 (page 279):

• The stability requirement for Cloud Amount (CA) and Cloud Top Temperature (CTT) was changed to 0.01 per decade and 0.25 K per decade, respectively.

• The accuracy (Bias) requirement for Cloud Effective Radius (CRE) was changed to 1 µm.

• The stability for Cloud Effective Radius (CRE) was changed to 1 µm per decade.

However, these recent changes have not been taken into account when developing the CLARA-A2.1 data record (i.e., the original CLARA-A2 CDR was completed before 2016) but they will be relevant for upcoming new CLARA-A editions.

2.1.4 Summary of target requirements (KPIs)

The CLARA-A2.1 cloud products are brokered from the CM SAF project, and consequently cannot be altered in C3S_312b_Lot1 and C3S2_312a_Lot1. Therefore, the current target requirements on the key performance of the data set, measured within C3S by the so-called Key Performance Indicators (KPIs), are defined as the requirements achieved (with original requirements described in D3) in previous CLARA-A2.1 validation activities in CM SAF. These values are listed in Table 2-2.

Table 2‑2: Key Performance Indicators (KPIs) or target requirements (i.e., fulfilled requirements for CLARA-A2.1 in the CM SAF project) for the cloud products CFC, CTP, LWP and IWP (monthly means) of interest for C3S_312b_Lot1. More details are given in reference D5 (Table 1.1 to Table 1.3).

For the evaluation of the ICDR, corresponding products from the MODIS instrument (MODIS Collection 6.1) are used as a reference. The distribution of the de-seasonalised differences between MODIS products and the CLARA.A2.1 TCDR has been compiled and the corresponding 2.5 and 97.5 percentile differences are given in Table 2-3.

Table 2‑3: KPI percentiles (2.5 % and 97.5 %) requirements for the CLARA-A2.1 CFC, CTP, LWP and IWP product differences against MODIS C6.1.

These percentiles are used to check, by means of a binomial test at 5 % significance level, whether the corresponding ICDR differences are consistent with the TCDR differences or not. Further details on these tests are found in the Report on Updated KPIs (D10).

2.1.5 Discussion of requirements with respect to GCOS and other requirements

Table 2-4 shows how the CLARA-A2.1 TCDR requirements compare to GCOS requirements. The achieved CLARA-A2.1 results in Table 2-4 are estimated at a product grid resolution of 0.25° which is approximately twice as fine as the GCOS-referred values. The reason for this is explained by requirements of higher spatial resolutions from other user categories than the climate research community (e.g., regional NWP and climate modelers, and the solar energy community). Despite the finer resolution, GCOS requirements are met for both CFC and CTP (at least when considering the slightly relaxed requirements for semi-transparent clouds). For LWP and IWP (both referring to the CWP requirement in Table 2-1) compliance is also good even if requirements are not expressed in the same way by GCOS as by CM SAF. Details of the validation results can be found in Karlsson et al. (2017). Regarding the GCOS requirement on the temporal resolution (interpreted as the basic required observation frequency for calculation of useful Level-3 products), the CLARA-A2.1 TCDR can only fulfil this requirement over the entire observation period at moderate to high latitudes. The tropical region will generally only be covered every 4^th to 8^th hour during the first two decades while later the observations frequency has increased with a maximum frequency better than 3 hours occurring around 2009.

Table 2‑4: GCOS-154 requirements for CFC, CTP, LWP and IWP compared to CLARA-A2.1 requirements.

2.1.6 Data format and content issues

The CLARA-A2.1 cloud property products are defined using standard data formats (netCDF) and map projections (regular latitude/longitude grids). Meta data definitions follow the Climate & Forecast conventions (http://cfconventions.org/).

2.2 Cloud properties TCDR ESA AATSR v3.0 + ICDR ESA SLSTR v3.1.1+v4.0

As the Cloud_cci products cover the same ECVs as the CLARA-A2.1 CDR, the requirements are very similar. In particular, the users of the data can be considered to be the same and the same GCOS requirements specified in Table 2-1 apply. Due to the consistent overpass time of (A)ATSR and SLSTR, the Cloud_cci can be considered to be a more stable and consistent record of Cloud Properties, and it contains more properties ¹ , than CLARA-A2.1. However, the Cloud_cci CDR is considerably shorter in time than the CLARA-A2.1 record and there is a four-year gap between the end of the TCDR and the SLSTR ICDR. Also, (A)ATSR provides sparser spatial coverage (due to its relatively narrow swath) compared to AVHRR. SLSTR has a wider swath than AATSR (although still narrower than AVHRR) and, from 2018 onwards, two SLSTR instruments flying with interleaved orbits provide global twice-daily coverage (with 10 am and 10 pm local-time overpasses).

The additional parameters provided in the Cloud_cci CDR over the CLARA-A2.1 products provide some additional scope for the application of the data. Particular points of interest include:

Cloud Optical Thickness (COT)

This is essentially a measure of the opacity of a cloud. Optically thick clouds will fully attenuate incoming radiation, resulting in purely diffuse solar radiation reaching the ground (transmitted through the cloud by multiple scattering) and thermal emission from the cloud top. Optically thin clouds will allow at least some radiation to pass through them without scattering or absorption. Thus, along with cloud height, optical depth is a primary controlling parameter in determining the radiative impact of clouds.

Cloud Effective Radius (CER)

CER is a measure of the size of the particles making up the cloud. This has an effect on the reflectivity of clouds (with smaller cloud particles being much more effective at reflecting solar radiation back into space). Furthermore, knowledge of the cloud droplet size provides insight into the cloud condensation nuclei which control the formation of the cloud in the first place, thus providing sensitivity to the impact of aerosols (both natural and anthropogenic) on Cloud Properties.

1. Although it should be noted that CLARA-A2.1 based COT and CER estimates are available from the CMSAF; they just aren’t included in the dataset brokered to C3S. ↩

2.2.1 Summary of target requirements (KPIs)

Similar to the CLARA-A2.1 product, Cloud_cci v3 is a brokered product from ESA’s CCI programme, and cannot be substantively changed for C3S_312b_Lot1 and C3S2_312a_Lot1. Thus, the KPIs for the Cloud_cci data are taken from the data quality and verification analysis performed within the Cloud_cci project itself. The resulting KPIs are summarized in Table 2-5.

Table 2‑5: Key Performance Indicators (KPIs) for the CFC, CTP, CTT, COT, CER, LWP and IWP products (monthly means) of Cloud_cci v3.0 of interest for C3S_312b_Lot1.

In order to monitor the performance of the SLSTR extension ICDR, new KPI values were also generated from comparisons of the Cloud Properties TCDR ESA AATSR to the NASA MODIS Cloud Properties product. These comparisons are represented as the 2.5 and 97.5 percentiles of the distribution of differences between (A)ATSR monthly-mean values and the corresponding MODIS values (corrected for the mean seasonal cycle), and are summarised in Table 2-6.

Table 2-6 : KPIs to be applied to corresponding SLSTR ICDR products in Table 2-4, based on comparison of the Cloud_cci v3.0 TCDR against MODIS Collection 6.1.

2.2.2 Data format and content issues

The Cloud_cci v3 Cloud Property products are defined using standard data formats (netCDF) and map projections (regular latitude/longitude grids). Meta data definitions follow the Climate & Forecast conventions (http://cfconventions.org/).

3. Gap Analysis

3.1 Description of past, current and future satellite coverage

3.1.1 Cloud Properties TCDR AVHRR CLARA v2.0 + ICDR v2.x

The CLARA-A2.1 TCDR is based on radiances measured by the AVHRR sensor. This sensor is carried by satellites travelling in sun-synchronous polar orbits with daytime equator crossing times either in the morning or in the afternoon. With the nominal constellation of at least two satellites (i.e., one morning and one afternoon satellite) this gives four observations per day at the equator while the observation frequency increases at higher latitudes (at maximum 28 observations per day at the poles) due to increasing overlaps between the satellite swaths. The sensor is basically a five-channel instrument but it has been available in three different versions with slightly modified channel compositions as illustrated in Table 3-1. This table covers all historically available satellites and AVHRR versions, i.e., not only those covered by CLARA TCDR v2.0. The missing satellites for the TCDR AVHRR CLARA v2.0 are Tiros-N, NOAA-6, NOAA-8, NOAA-10 and Metop-C. Note that the third version of the instrument (AVHRR/3) is capable of switching between two different channels in the short-wave infrared spectral region (i.e. channels 3A and 3B). The periods when switching was actually carried out for the data used in TCDR AVHRR CLARA v2.0 are presented in Table 3-2. Note that data from Metop-B was not used for the TCDR in the years 2016-2018 due to errors in the calibration of visible channels. However, both Metop-B and Metop-C data are used in the ICDR data record after getting access to updated calibration data.

Table 3‑1: Spectral channels of the Advanced Very High Resolution Radiometer (AVHRR). The three different versions of the instrument are described as well as their corresponding satellites. Notice that channel 3A was only used continuously on NOAA-17 and on the Metop satellites. For the other satellites with AVHRR/3 it was used only for shorter periods. (Table taken from reference document D4 but extended with information on satellites Tiros-N and Metop-C).

Table 3‑2: Channel 3A and 3B activity for the AVHRR/3 instruments during daytime. Notice that the given time periods show the availability in the CLARA TCDR v2.0 and not the true lifetime of the individual sensor/satellite. The table is taken from reference document D4 (slightly modified with respect to end of data record).

In the following text we will refer to some figures prepared for the description of the complete CLARA-A2.1 data record from the CM SAF project (described in D4). It is identical to the TCDR AVHRR CLARA v2.0 except that it also includes data for the first six months of 2019.

Figure 3-1 shows an overview of all NOAA satellites with AVHRR measurements which were used for the compilation of the CLARA-A2.1 CDR. This figure also shows how observation times have changed due to various degrees of orbital drift for the individual satellites. Another prominent feature seen in Figure 3-1 is the temporal observation inhomogeneity. For example, initially only one (afternoon) satellite was used while during the last ten years of the series 4-5 individual satellites could be used in parallel.

Figure 3‑1: Local solar times at equator observations for all satellites from NOAA-7 to NOAA-19 and Metop A/B. Shown are all data that were used for the CLARA-A2.1 processing. Notice that the figure shows both ascending (northbound) and descending (southbound) daytime equator crossing times. Each satellite has also another equator crossing occurring 12 hours later (at night or in the evening). (Figure taken from reference document D4).

The lack of morning satellite data in the beginning of the period is explained by the retrieval algorithm requirement to have all 5 spectral AVHRR bands available. AVHRR/1 has only 4 spectral bands, thus lacking the split-window channel at 12 µm being crucial for, e.g., thin Cirrus detection and accurate cloud top height determination.

The variable coverage of AVHRR observations over the CLARA-A2.1 time period is problematic in the sense that it violates the requirement of having as homogeneous observations as possible when compiling CDRs. However, it can be noticed that the inhomogeneity concerns especially the satellites in the morning orbit while observations from the afternoon orbit have adequate coverage over the entire time period. Thus, meaningful studies of e.g. climate trends are still possible if restricting the study to the use of data from afternoon satellites (as demonstrated by Karlsson and Devasthale, 2018 and by Stengel et al., 2020).

If looking a bit closer on the actual data coverage, it is clear that some satellites have experienced technical problems of varying kinds (e.g., scan motor problems, instrument noise in the AVHRR 3.7 micron channel, etc.) which have led to data losses or partially corrupt scenes. Noise problems in the 3.7 micron channel affected all satellites carrying AVHRR/1 and AVHRR/2 sensors. Problems were normally weak or moderate but some satellites experienced particularly high and severe noise levels during some periods of their lifetime (e.g., NOAA-7, NOAA-9, NOAA-10 and NOAA-12). The periods of high noise levels lead typically to overestimated cloud amounts from most of the used cloud screening methods. For example, for the CLARA-A2.1 predecessor CLARA-A1, it contributed to creating a large negative trend in cloud cover because of overestimated cloud amounts for the early satellites in the series of used satellites (Sun et al., 2015). A noise filtering procedure was introduced for CLARA-A2.1 (Karlsson et al., 2017) which reduced the impact of excessive 3.7 micron noise.

Serious gaps in data coverage have also affected the AVHRR observation series. The largest gap occurred between September 1994 (loss of NOAA-11) and April 1995 (launch of NOAA-14). A few other gaps also exist but normally covering less than a month´s worth of data. Problems with partially corrupt scenes have also temporarily led to a decrease of the frequency of available observations. Corrupt scenes (or scenes with a sub-set of corrupt scan lines) are normally associated with scan motor problems and may occur for all satellites, and especially towards the end of the AVHRR instrument life time. A particularly problematic period occurred in the period 2000-2003 when several satellites had problems with corrupt scan lines from AVHRR. This was challenging for various parameter retrieval schemes since not all corrupt scan lines in these scenes were correctly labelled as corrupt in the original Level 1b files. Therefore, in the CLARA-A2.1 case, large efforts were undertaken to secure blacklisting of corrupt lines while at the same time using as much as possible of remaining non-affected scan lines.

The extension of a TCDR (in this case extending CLARA-A2 with three additional years 2016-2018) and the introduction of ICDR production after the end of the official length of the TCDR period (2019+) are not trivial to carry out since some of the necessary input data may be lacking or be restricted during the extension period. This is one of the reasons why this type of extension is called “Intermediate” (i.e., being produced with some restrictions compared to the original TCDR) while awaiting the next reprocessing effort (i.e., CLARA-A3).

For the CLARA-A2.1 case, a particular problem occurred regarding the access to a stable calibration of visible channels which takes into account the degradation with time of the sensor response. The original CLARA-A2 TCDR relied upon the updated results provided by NOAA following the original approach by Heidinger et al. (2010). Unfortunately, the production of these updated calibration results from this particular method ceased after 2016. It means that all results from 2017 and onwards are based on extrapolated calibration data which are not adjusted after access to more recent calibration reference data. Since these calibration corrections have non-linear terms which are not very well determined for the youngest satellites, deviations may grow in an unrealistic way with time. This was the case for the visible AVHRR radiances from the Metop-B satellite and it was clear that this affected the derived CLARA products in a negative way. Consequently, it was decided to exclude Metop-B data from CLARA-A2.1 for the extension period 2016-2018. However, updates of the Metop-B and Metop-C calibration have been received recently (2020) which means that data from these satellites is now included in the ICDR data records.

The CLARA observation period can be extended both back in time (until 1978, see Figure 3-2 and Figure 3-3) and forward in time taking into account satellites that are in operation from 2019 until now. Such an extension is planned in the CM SAF project and this new CLARA version will be denoted CLARA-A3. The new version will tentatively cover the time period 1978-2020, i.e. almost 42 years (in particular, covering the new reference climatology period 1991-2020). It will partly solve the observation inhomogeneity seen for CLARA-A2 in the beginning of the period by adding more satellites with the AVHRR/1 sensor. However, this will also require handling of new potential inhomogeneities due to the inclusion of a more restricted AVHRR instrument. Retrievals based on AVHRR/1 are expected to lead to slightly decreased cloud detection capabilities (especially concerning thin Cirrus clouds) and to some degraded cloud top height estimation accuracy (again, especially for high thin clouds).

As already mentioned, the backward extension will solve the lack of morning satellite data in the CLARA-A2.1 dataset in the period 1982-1991. The missing data will then be provided by the morning satellites NOAA-6, NOAA-8 and NOAA-10. Regarding the earliest period 1978-1981, a full coverage with both morning and afternoon satellites is unfortunately not possible. Afternoon observations will start in October 1978 with the Tiros-N satellite which is then later replaced by NOAA-7 in June 1981. However, a data gap of several months (March until May) will exist in 1981. Similarly, morning observations will start in June 1979 with the NOAA-6 satellite and it is later replaced by NOAA-8 in March 1983. It means that for the first half year with afternoon observations (i.e., October 1978 until May 1979) morning observations will be missing.

Figure 3‑2: Overview of all NOAA satellites (except NOAA-19 launched in 2009) carrying the AVHRR instrument. Notice that the figure only shows expected operational life and not the true operation periods for individual AVHRR instruments. Figure 3-3 gives a more complete picture with all AVHRR-carrying satellites (including EUMETSAT Metop satellites). (From https://www.nesdis.noaa.gov/content/noaa-15-makes-100-thousandth-orbit).

Figure 3‑3: Overview of all satellites carrying the AVHRR instrument on individual satellites from Tiros-N in 1978 until 2022. Vertical lines mark new launches or end of operations for individual AVHRRs (From EUMETSAT, 2023)

CLARA-A3 will most likely be the last TCDR based exclusively on data from the AVHRR instrument. The very last satellite carrying the AVHRR sensor is Metop-C which was launched on 7^th of November 2018. With a nominal expected lifetime of approximately 5 years it means that AVHRR data beyond 2025 (or maybe 2030 considering the extended life of most recent AVHRRs) will most likely not be available anymore. Of importance here is also that AVHRR observations from a satellite in afternoon orbit will cease in 2022 (according to NOAA). The last satellite carrying AVHRR in an afternoon orbit is NOAA-19. This satellite has now been operational for more than 12 years and it is thus approaching the end of its lifetime. A particular problem with the still available NOAA satellites in afternoon orbit carrying the AVHRR sensor is that orbital drift has already affected them seriously. In fact, for the last few years of the planned CLARA-A3 TCDR (at least years 2019 and 2020) these observations are better characterized as morning observations than afternoon observations as a consequence of the orbital drift. This is another reason for putting a definite end of exclusively AVHRR-based TCDRs to the year 2020.

The AVHRR-based CDRs covering more than four decades are unique and precious datasets from the climate monitoring point of view in that they represent the longest available multispectral imaging observation record from space. Possibilities to extend the observation series even further by use of data from other sensors would therefore be extremely valuable. The prospects here are good considering that the next generation polar-orbiting meteorological satellites will carry sensors with AVHRR-heritage spectral bands as a subset of their full range of spectral bands. Some of these satellites are already in orbit. The most important sensors in this respect are the Visible/Infrared Imager Radiometer Suite (VIIRS) and the Meteorological Imager (METimage) sensors. Both sensors’ spectral channels are described in Table 3-3 and Table 3-4, respectively. We note in Table 3-3 that heritage AVHRR channels are channels M5, M7, M10, M12, M15 and M16 for VIIRS. For METimage in Table 3-4, AVHRR-heritage channels are available at wavelengths 670 nm, 865 nm, 1630 nm, 3740 nm, 10790 nm and 12020 nm. Similar information about METimage is given by the following link: https://www.eumetsat.int/eps-sg-metimage.

Table 3-5 lists the current and planned satellites (including their expected lifetimes) carrying the VIIRS and METimage sensors. The VIIRS sensor is carried on the next generation US polar orbiting meteorological satellites following the NOAA-19 satellite. The first satellite with VIIRS was Suomi-NPP which was launched in 2011. Suomi-NPP was a precursor to the US part of the operational Joint Polar Satellite System program (JPSS). The first JPSS satellite (JPSS-1) was launched in 2017 and it was later renamed to NOAA-20. The US part of JPPS deals with satellites in the afternoon orbit. The second leg, i.e. the morning satellites, will be provided by EUMETSAT with the upcoming EPS-SG satellites. These satellites will carry the METimage sensor.

It should be noted that VIIRS observations in a stable afternoon orbit have been available since 2011. This means that VIIRS data can not only be used to extend CLARA-A3 beyond 2020 but it can also be used to replace data from NOAA-18 and NOAA-19 in the period 2011-2020 when the latter satellites were exposed to serious orbital drift. In other words, VIIRS data can be used to restore the nominal two orbit constellation of observations from AVHRR during the last decade of CLARA-A3 coverage. It means that VIIRS data can not only extend CLARA-A3 results but also improve CLARA-A3 results for the last decade of its temporal coverage. This possibility has been taken up in the planning of a further extension, named CLARA-A3.5, of the CLARA data record in the next CM SAF project phase covering the years 2022-2027. CLARA-A3.5 will then add results derived from the AVHRR-heritage channels of VIIRS from Suomi-NPP and NOAA-20. CLARA-A3.5 is scheduled for release in 2026.

Table 3‑3: Spectral channels of the Visible/Infrared Imager Radiometer Suite (VIIRS) sensor carried by the Suomi-NPP and JPSS satellites. (From https://weather.msfc.nasa.gov/sport/jpsspg/viirs.html).

Table 3‑4: Spectral channels of the Meteorological Imager (METimage) sensor to be carried on the EPS-SG satellites. (From DLR METimage presentation at 3rd Post-EPS User Workshop 29 September 2011)

Table 3‑5: Next generation polar orbiting meteorological satellites in low earth orbit (LEO) for the US/European Joint Polar Satellite System. Satellites are listed in chronological order based on true and planned launch dates. All information is taken from the WMO OSCAR site (https://www.wmo-sat.info/oscar/satellites/) in April 2022.

Table 3-5 shows that AVHRR-heritage information from one morning and one afternoon orbit will be provided for at least 20 more years from present time (i.e., nominally until 2045. Consequently, it means that CLARA-type CDRs will be possible to compile over a time period exceeding 60 years from the initial launch year of 1978 for the first AVHRR instrument.

It is finally worth mentioning that the Chinese Feng-Yun 3 satellites FY-3D to FY-3H carry the 25-channel VIS-IR spectrometer MERSI-2, which also includes all AVHRR heritage channels (see https://www.wmo-sat.info/oscar/instruments/view/279). Partly, these satellites fly in similar orbits to the JPSS and EPS-SG series. Thus, in case problems are encountered with near future satellite coverage of the satellites in the US/European Joint Polar Satellite System, data from the Feng-Yun satellites can be used as a backup solution provided that calibration homogenization can be achieved. However, it should be noted that the upcoming FY-3E and FY-3H satellites will be operated in an early-morning orbit with an equatorial overpass time of 6:00 am.

3.1.2 Cloud Properties TCDR ESA AATSR v3.0 + ICDR ESA SLSTR v3.1.1+v4.0

The Cloud_cci v3.0 is based on radiances provided by the ATSR series of sensors. These instruments flew on sun-synchronous polar orbiting satellites with daytime equatorial crossing times in the mid-morning: 10:30 Local Time on Descending Node (LTDN) for ATSR-2 and 10:00 LTDN for ENVISAT, with both satellites sharing the same ground track. There were 14.3 orbits per day, meaning 28 equatorial overpasses per-day, with measurements covering a total of 18% of equatorial circumference of the Earth (with equally spaced 512 km swaths). The observation frequency increased at higher latitudes (with a maximum of 14 observations per day at the poles) due to increasing overlaps between the satellite swaths. Both sensors provided the same seven channels (and used the same conical dual-viewing geometry), but not all channels were provided at all times, or at full digitization rate, from ATSR-2, due to limitations of the data bandwidth provided by the ERS-2 platform. Over ocean regions, ATRS visible channels were often only provided in a 256 pixel “narrow-swath” mode. The channels provided by both instruments were centred at 0.55, 0.67, 0.87, 1.6, 3.7, 10.8, 12.0 \( \mu m \) and the filter band passes were very similar between instruments. Despite the low-data rate modes of ATSR-2, the combination of the very similar instrument specifications, very close orbital parameters and the lack of any significant orbital drift in the ERS-2 and ENVISAT satellites mean that ATSR-2 and AATSR provide a highly consistent data record, especially when compared to that provided by the AVHRR record used by the CLARA-A2.1 TCDR (although AVHRR provides a much longer data record).

The current TCDR from Cloud_cci v3.0 begins with the launch of ERS-2 in mid-1995 and continues until the failure of ENVISAT in April 2012. Due to instrument problems, there is a six-month data gap in the ATSR-2 record from January to June 1996.

There is an overlap of 1 year of data between the two platforms, between mid-2002 (when ENVISAT was launched) and mid-2003 (when the onboard data storage on ERS-2 failed). There is additional ATSR-2 data available up-to 2009, but this is not global as data could only be collected when the satellite was within line-of-sight with a ground receiving station, and has not been included in the TCDR. There is some scope to push the coverage of the ATSR cloud record back to 1991, by using the ATSR-1 instrument (onboard ERS-1), which also flew in a similar orbit to its successors. However, ATSR-1 lacked the shortwave channels (apart from the 1.6 \( \mu \) m channel), which would reduce the information available to daylight retrievals and would represent a significant inhomogeneity in the TCDR.

The extension of the ATSR TCDR relies on the SLSTR sensors onboard the Sentinel-3 platform. SLSTR represents a significant upgrade over (A)ATSR, providing a wider swath, two satellites with interleaved orbit swaths, additional channels and the data availability security of an operational system. The Sentinel-3s have a very similar orbit to ENVISAT and the ERS satellites, with a sun-synchronous orbit with an LTDN of 10:00, and 14.3 orbits per-day. However, there is slightly longer than 4-year gap between the end of the (A)ATSR record and the first SLSTR data. There are several options available to fill this gap, as ORAC can be applied to most radiometers with similar channels to those provided by ATSR. Indeed, cloud CDRs of ORAC applied to both MODIS and AVHRR already exist, having been produced in the Cloud_cci program. Furthermore, the CLARA-A2.1 product also discussed in this document also provides a comparable product which fills the (A)ATSR-SLSTR data gap.

It should also be noted that ORAC could be applied to the VIIRS and METimage instruments described above, which could compliment the SLSTR ICDR.

3.2 Development of processing algorithms

3.2.1 Cloud Properties TCDR AVHRR CLARA v2.0 + ICDR v2.x

3.2.1.1 General aspects common to all selected CLARA products

A fundamental aspect is that TCDRs based on AVHRR data must be based on an accurate AVHRR Fundamental Climate Data Record (FCDR), i.e., quality checked radiances corrected for degradations and with proper inter-calibration between individual satellites. The work with FCDRs is an ever-ongoing task (e.g., new data added to the series must be analysed, inter-calibrated and appropriately corrected for possible degradations).

Another factor that affects all CLARA products (as well as all other available AVHRR-based CDRs) is the inhomogeneity of the observational record, i.e., mainly the different observation frequency in the beginning and in the end of the observational record. Orbital drift effects also contribute but the main problem is the lack of morning observations in the current CLARA-A2.1 CDR in the initial part of the covered period. In the worst case, this may lead to artificial trends in the CDR as discussed by Karlsson et al. (2018). A way to mitigate these problems is to add AVHRR/1 observations to the used observational record (as mentioned in the previous section). For the extension of the CDR beyond the AVHRR era, adaptations have also to be done to cope with the AVHRR-heritage channels on post-AVHRR sensors.

The two mentioned aspects are by far the most important ones for assuring a high quality of an AVHRR-based TCDR. The following sub-sections will provide some more details on these aspects. There are also issues (limitations) with the current retrieval algorithms but these will be discussed more in detail later in sections 3.4 and 3.5.

3.2.1.1.1 Maintaining development of the AVHRR Fundamental Climate Data Record

The AVHRR FCDR used as a basis for CLARA-A2.1 processing was only inter-calibrated and corrected for sensor degradation for the visible channel 1, the near-infrared channel 2 and the short-wave infrared channel 3a (see Table 3-1). This was based on an extension of the method originally proposed by Heidinger et al. (2010). Infrared radiances were still based on the nominal operational NOAA calibration method utilizing onboard black body reference sensors.

In order to coordinate development and to improve progress in calibration methodologies, a special project (SCM-5: Advancing the AVHRR FCDR) was defined in 2014 within the framework of the SCOPE-CM (Sustained and COordinated Processing of Environmental satellite data for Climate Monitoring) activity - https://www.scope-cm.org/). SCOPE-CM is a high-level project network with links to entities like WMO, WCRP, GCOS, CGMS, CEOS and GEO. SCM-5 involved scientists from NOAA and NASA, and also from the CM SAF and ESA Cloud_cci projects. It was also linked to the EU-project FIDUCEO (Fidelity and uncertainty in climate data records from Earth Observations - https://research.reading.ac.uk/fiduceo/) where an improved infrared calibration for AVHRR data was under development. SCM-5 formally ended in February 2019 when SCOPE-CM entered a new project phase (Phase III).

The planning of the next CLARA-A3 CDR will utilize the outcome of SCM-5 which so far mainly consists of an upgraded calibration method for visible channels as a result of a joint NOAA/NASA effort. The infrared calibration upgrade from the FIDUCEO project could unfortunately not be finalized and only a preliminary but incomplete (i.e., only covering about 20 years of the record) version became available in August 2018 for testing. The preliminary FIDUCEO FCDR has recently been evaluated in the CM SAF project (CM SAF, 2022). It was concluded that the FCDR quality was acceptable for the latest AVHRR/3 generation of the sensor but that large problems were found related to the harmonization of results from the previous AVHRR/2 generation.

Continued coordination of AVHRR FCDR developments beyond SCM-5 is needed to ensure the success of future reprocessing events based on the full AVHRR data record. This concerns monitoring and correction of radiances from current and future satellites still equipped with the AVHRR sensor (mainly the NOAA-19 satellite and the Metop satellites) and complementing and extending the FIDUCEO infrared calibration work for the earliest satellites (not included in FIDUCEO coverage). A first step in this direction has been taken by EUMETSAT by forming a joint EUMETSAT/CM SAF project (named the AVHRR GAC FDR project) to build upon previous experiences of the work in CM SAF, ESA Cloud_cci and FIDUCEO for improving the AVHRR GAC data record. The outcome of this project will be an official release of the AVHRR GAC FDR which also will be the basis for the next edition of CLARA (CLARA-A3). This FDR will be released in 2023, almost simultaneously with the CLARA-A3 release. A continuation of the AVHRR GAC FDR project is foreseen by EUMETSAT and the goal is to try to release a full FCDR version by 2027.

3.2.1.1.2 Adaptations to AVHRR/1.

Adding AVHRR/1 data to the AVHRR record processed by CLARA-A3 or similar CDRs means that most retrieval methods, previously used to process AVHRR/2 and AVHRR/3 data, have to be updated to cope with an AVHRR sensor with reduced capability (i.e., not including a split-window channel at 12 µm). Such adaptations have now been completed in the CM SAF project in the preparation of the CLARA-A3 CDR.

3.2.1.1.3 Adaptations to AVHRR-heritage channels on post-AVHRR sensors VIIRS and METimage

A challenge for future CDRs (beyond CLARA-A3) will be the inclusion of radiances from AVHRR-heritage channels from VIIRS and METimage. This requires that appropriate measures are taken to remove effects of differences in spectral response functions between original AVHRR channels and the AVHRR-heritage channels on the new sensors. The largest differences here are found for AVHRR channel 2 with central wavelength at 0.9 µm but with a very wide spectral response (see Table 3-1) compared to corresponding channels on the new sensors. However, data from this channel is only used marginally by the CLARA-A2.1 cloud masking method (affecting CFC) and not at all for the other cloud products. Consequently, adaptations and corrections here appear not to be crucial for securing the quality of the CLARA products in the future. A first step towards the additional use of non-AVHRR sensors in the CLARA dataset will be taken shortly to form the CLARA-A3.5 data record where AVHRR-heritage channels in VIIRS will be used to improve the coverage of observations from the afternoon orbit for the period 2012-2024. CLARA-A3.5 is planned for release in 2026. The notation 3.5 indicates that this is not a completely new CLARA edition but, at the same time, it is not just an extension since it involves some development work to define the AVHRR-heritage channels from VIIRS.

Another necessary adaptation concerns the improved horizontal resolution of the original dataset from VIIRS and METimage sensors. Both sensors provide finer resolutions (better than 1 km) than the AVHRR instrument. In addition, the VIIRS instrument is using a quite advanced across-track scanning mechanism to maintain a high horizontal resolution over the full swath width. In conclusion, efforts are required to resample VIIRS and METimage data to be compatible with the original 5 km AVHRR GAC dataset. Responsible agencies (NOAA and EUMETSAT) have to be aware of the need for such a resampled radiance dataset in order to allow a proper extension of the AVHRR-based CDRs forward in time. To extend the CDR with original VIIRS and METimage data with native spectral and spatial resolution would risk creating unwanted discontinuities in the results when shifting from AVHRR to AVHRR-heritage sensors.

Initial efforts of the ISCCP team of NOAA/NCEI has led to the definition of a global VIIRS dataset with all 16 M-Bands (see Table 3-3) with a horizontal resolution similar to GAC. This dataset is denoted VGAC (VIIRS data resampled to AVHRR GAC resolution). A commitment to produce a complete VGAC dataset (covering historic, present and future VIIRS data) for ISCCP and for other purposes has been announced by NOAA/NCEI (Andy Heidinger, NOAA, personal communication). The VGAC dataset will be used for the previously mentioned CLARA-A3.5 data record.

3.2.2 Cloud properties TCDR ESA AATSR v3.0 + ICDR ESA SLSTR v3.1,1+v4.0

As with the CLARA CDRs, the stability and quality of the input data is the key parameter which influences the reliability of the Cloud_cci v3 CDRs. The current AATSR TCDR is based on version 3 of the “AATSR multimission archive” maintained by CEDA and the UK National Earth Observation Data Centre (NEODC). This record incorporates the latest calibration corrections (including long-term drift corrections from vicarious calibration) and represents the most consistent and accurate record of radiances from the (A)ATSR record. A future update to this record would make a reprocessing of the Cloud_cci TCDR possible.

In the case of the SLSTR ICDR, the status of the level 1 radiances is considerably less stable. As of yet, there is no consistently processed version of the SLSTR radiance record, as the level 1 processing has been under development while the data has been in production. Thus, data from early in the SLSTR record has considerably worse calibration and geolocation than more recent data.

EUMETSAT has provided updated calibration corrections to SLSTR shortwave channels, communicated through the Sentinel-3 Scientific Validation Team (S3VT), which have been applied retrospectively. However, it is only very recently that work has begun on characterizing the stability of the SLSTR sensors and determining drift corrections for their calibration (as has been done for the (A)ATSR instruments). Therefore, no drift correction has been applied to the SLSTR data used for C3S.

3.2.2.1 Adaptions of the ORAC scheme to better exploit SLSTR

As mentioned above, SLSTR provides some additional channels over the earlier AATSR instruments. Of particular note is the new 1.3 \( \mu \) m channel, which, due to its location in a water-vapour absorption feature, is particularly sensitive to the presence of high-altitude clouds. Utilizing this channel in the retrieval scheme itself is unlikely to be beneficial, as accurate knowledge of the water vapour profile is needed to accurately model the radiances. However, the use of this channel in prior cloud-detection and characterization has been studied under the Cloud CCI+ project (which is a follow-on project to the original ESA-CLOUD-CCI project) and an assessment of its impact on the quality of ORAC cloud retrievals is underway.

3.2.2.2 Forward model improvements

Further improvements to the forward modelling of clouds for the ORAC retrieval scheme are also underway. In particular:

• The SLSTR ICDR makes use of ERA-5, rather than the ERA-Interim used for the TCDR.

• The spectral dependence of cloud scattering and absorption will be modelled across the bandpass of the instrument channels (rather than at the channel centre as was previously the case).

• At present cloud is modelled as an infinitesimally thin layer within an atmosphere modelled by RTTOV. The modelling of cloud geometric thickness effects will also be investigated in the ongoing Cloud CCI+ project.

• The use of new ice cloud optical properties will also be investigated, as these become available.

• Improvements in the propagation of uncertainty from L2 products to gridded L3 products is also under investigation.

3.3 Methods for estimating uncertainties

3.3.1 Cloud Properties TCDR AVHRR CLARA v2.0 + ICDR v2.x

In the following sub-sections, the uncertainty estimations are described product by product. However, generally CLARA-A3 uncertainty estimates will be expanded by considering uncertainty in ancillary data used in the retrieval process, incl. surface albedo, emissivity and temperature and atmospheric (trace) gas amounts. Furthermore, the aggregation to Level 3 will be refined according to methods described in Stengel et al. (2017).

3.3.1.1 Cloud Fractional Cover (CFC)

The current CLARA-A2.1 CFC products are not associated with any uncertainty estimates. Thus, uncertainty information is only available as results achieved by associated validation activities. Some quality flagging is applied in the original PPS cloud masking product but this information is not propagated or compiled for the Level 3 products.

For CLARA-A3, Bayesian cloud masking will be used which enables estimation of the uncertainty of cloud masking based on the used spectral information. However, this uncertainty will not consider uncertainty in input data (AVHRR radiances or auxiliary data). Further work is needed here (e.g., Monte Carlo simulations) for potential CLARA editions beyond CLARA-A3.

3.3.1.2 Cloud Top Pressure (CTP)

The current CLARA-A2.1 CTP product is not associated with any uncertainty estimates. Thus, uncertainty information is only provided as results achieved by associated validation activities. One aspect leading to additional uncertainty and an inconsistency with the CFC product is the fact that not all pixels declared cloudy in the basic PPS cloud mask product will be given a valid CTP result. This concerns very thin high clouds and some fractional (sub-pixel) clouds at all altitudes. Pixels with lacking CTP results amount to about 2-3 % of all pixels declared cloudy.

For CLARA-A3 limited uncertainty information (i.e., percentile distributions) will be provided in original Level 2 products and this can be utilized to estimate the uncertainty of Level 3 products. Also, all cloudy pixels will be associated with a valid CTP result in CLARA-A3.

3.3.1.3 Liquid Water Path (LWP)

LWP in CLARA-A2.1 is accompanied with an estimate of the uncertainty. This estimate is derived by propagating uncertainties in level-1 reflectance to τ and r_e, and finally to LWP. Hence, only one source of uncertainty (i.e. in the satellite measurements) is taken into account. Uncertainties are aggregated to Level 3 by linear averaging.

3.3.1.4 Ice Water Path (IWP)

Uncertainty estimates for IWP in CLARA-A2.1 and CLARA-A3 are/will be derived in the same way as for LWP.

3.3.2 Cloud Properties TCDR ESA AATSR v3.0 + ICDR ESA SLSTR v3.x

As the ORAC retrieval system is based on optimal estimation, rigorous propagation of uncertainty into L2 retrieval products is integral to the scheme. Under the Cloud_cci+ project, the input uncertainty budget used by the retrieval scheme has been extensively revamped, including improvements in the handling of instrument noise and uncertainties in ancillary data used by the retrieval scheme. Although these developments did not reach a sufficient level of maturity to be included into ICDR production for C3S, they should be incorporated into future reprocessing of the SLSTR record.

The propagation and characterization of uncertainty in Cloud_cci products, both L2 and L3, is detailed in the ESA Cloud_cci Comprehensive Error Characterisation Report (CECR) (Stengel et al. 2018), and is briefly described here.

3.3.2.1 Cloud Fractional Cover (CFC)

Current ESA Cloud_cci products do not provide uncertainty information on the cloud fraction. The retrieval scheme uses a neural-network approach to flagging cloud pixels from L1 radiances, which does provide an uncertainty estimate based on how well the neural network is able to classify the radiances. However, it is not straightforward to map this value to a quantitative estimate of the uncertainty in cloud cover.

The Incorporation of uncertainty into CFC products from ORAC remains as a task for future versions of the processing chain.

3.3.2.2 Cloud Top Pressure (CTP), Height (CTH) and Temperature (CTT)

In the scheme used to produce Cloud_cci and initial SLSTR ICDR products, the uncertainty of input L1 radiances as well as factors accounting for scene inhomogeneity are propagated to give L2 uncertainty. In the case of CTP (or CTH, CTT), this has been shown to underestimate the true uncertainty in the retrieved product when compared to co-located observations by the CALIOP lidar (see Stengel et al. 2018). However, the distribution of uncertainty conforms reasonably closely to the expected Gaussian shape, at least for situations where the retrieval has provided a good fit to the measured radiances.

3.3.2.3 Cloud Optical Thickness (COT), Effective Radius (CER) and Water Path (LWP, IWP)

The error propagation and uncertainty estimation of these parameters are identical to (and simultaneous with) that of CTP, however validation of the resulting uncertainty is not readily achievable.

3.3.2.4 Propagation of uncertainty to gridded L3 products

C3S products from Cloud_cci data and the SLSTR ICDR follow the methodology described by Stengel et al. (2018), with four separate uncertainty parameters being included for each mean parameter, x:

3.4 Opportunities to improve quality and fitness-for-purpose of the CDRs

3.4.1 Cloud Properties TCDR AVHRR CLARA v2.0 + ICDR v2.x

Sections 3.2 and 3.3 have already indicated some of the improvements of CLARA cloud CDRs which will be included in the upcoming CLARA-A3 CDR. The most prominent updates here are the introduction of Bayesian (probabilistic) cloud masking (Karlsson et al., 2015, 2020) and CTP retrievals based on artificial neural networks (Håkansson et al., 2018). These changes are largely made possible by the access to high-quality cloud information from the CALIPSO-CALIOP sensor since 2006 (Winker et al., 2009).

The high potential of using artificial neural network methods for improving the cloud product quality is not only demonstrated for the planned CLARA-A3 CTP product but also for products like cloud masking and cloud phase retrievals (Stengel et al., 2017, 2020 and Sus et al., 2018). Also, this development is a consequence of the availability of the CALIPSO-CALIOP observations. Continuing development efforts here (e.g. within the ESA CCI+ project) will most likely impact future compilation of AVHRR-based CDRs. A central question here will be to find out if ANN-based methods are able to match physical retrieval methods (e.g., as described by McGarragh et al., 2018).

An interesting extension of the AVHRR FCDR was proposed by NOAA in the SCM-5 project. NOAA proposed to add selected information from the High-resolution Infrared Radiation Sounder (HIRS) instrument to the AVHRR radiance dataset in order to improve in particular the cloud top pressure retrieval. HIRS is another instrument which has been flying on the same satellite platforms as the AVHRR instrument historically but for the purpose of providing coarse resolution vertical sounding information. Current AVHRR-based cloud top retrieval methods have been shown to largely underestimate cloud top heights of thin cirrus clouds. At the same time, it is well-known that the HIRS measurements are very sensitive to the existence of thin Cirrus clouds. Thus, to merge selected HIRS information (e.g., channels near 7 µm and 13 µm) with AVHRR data would potentially improve both thin Cirrus detection and the retrieval of cloud top heights for thin Cirrus clouds. A fusion method applicable for sensors like HIRS and AVHRR has been proposed by Weisz et al. (2017) and the method will be applied in the next PATMOS-X dataset (as mentioned by Heidinger et al., 2014). However, the proposed fusion technique is not the only possible approach here so further research is needed to identify the most efficient method to merge information from the two sensors. In addition, just as for AVHRR-heritage applications, methods must be developed to cope with the transfer from the HIRS sounding instrument to corresponding HIRS-heritage instruments on the JPSS and EPS-SG satellites to enable a consistent use over the full time period of possible AVHRR and AVHRR-heritage measurements.

3.4.2 Cloud properties TCDR ESA AATSR v3.0 + ICDR ESA SLSTR v3.1.1+v4.0

The planned development of the ORAC retrieval scheme, as applied to (A)ATSR and SLSTR, has already been described in section 3.2.2. ORAC is under active development, both through the ESA CLOUD CCI+ program and through national UK funding (in particular, under the National Centre for Earth Observation). New improvements of the scheme, where applicable, will be fed through to the production of improved CDR products from SLSTR.

It is also worth noting that the ORAC scheme is not specifically designed for application to (A)ATSR or SLSTR. CDRs have already been produced using the scheme for the AVHRR and MODIS instruments, under previous iterations of the CCI programme. The scheme has also been applied to geo-stationary sensors (SEVIRI, GOES and Himawari-AHI), and improved application of the scheme to SEVIRI in particular (making use of the water-vapor sounding channels provided by the instrument) is being undertaken in CLOUD CCI+.

The code includes the ability to utilize sounding channels (CO₂ slicing and water-vapor absorption), as well as a multi-layer cloud retrieval mode (Poulsen et al., 2018), which greatly improve on the shortcomings of the existing “heritage channel” (AVHRR-like) CDRs produced in Cloud_cci. Furthermore, it retrieves the properties of dual-layer cloud scenes. Thus, the scheme provides the scope for the production of cutting-edge CDRs from a wide range of instruments, all with a consistent retrieval approach.

3.5 Scientific Research needs

3.5.1 Cloud Properties TCDR AVHRR CLARA v2.0 + ICDR v2.x

Regarding the development of processing algorithms, further improvement in this field will be linked to the availability of high-quality reference measurements from active sensors, like the CALIPSO-CALIOP sensor. As the access to such reference data is not guaranteed (e.g., by further delay or problems with data from the upcoming EarthCARE mission), development may be slowed down. This is especially serious for sensors like METimage which most likely will not have any overlap with CALIPSO-CALIOP data (in contrast to VIIRS on Suomi-NPP and NOAA-20, where we have more than 10 years of useful CALIPSO data).

If the reference data availability is guaranteed, the next question is whether Bayesian cloud masking methods and/or physical retrieval methods including proper error propagation modelling (e.g., McGarragh et al., 2018) will be able to cope with the results provided by ANN-based methods. The latter have shown remarkable progress in generating various high-quality cloud products but they still lack appropriate methods for estimating the uncertainty of the products. However, promising attempts (e.g., Pfreundschuh et al., 2018) to address this problem are under consideration. Intensive research in this field is expected to continue.

Finally, a limiting factor for all climate monitoring activities from space is the very high requirement on radiometric calibration in order to be able to distinguish a climate trend from trends caused by instrumental inhomogeneities, errors or noise. Further work on achieving high quality radiance references is therefore needed in addition to securing the availability of active cloud sensing instruments in space.

3.5.2 Cloud Properties TCDR ESA AATSR v3.0 + ICDR ESA SLSTR v3.x

The requirements for the further improvements of the (A)ATSR and SLSTR CDRs are identical to those for the AVHRR CLARA CDRs.

3.6 Opportunities from exploiting the Sentinels and any other relevant satellite

3.6.1 Cloud Properties TCDR AVHRR CLARA v2.0 + ICDR v2.x

The Sentinel-3 mission carries the SLSTR and OLCI instruments which both have several AVHRR-heritage channels. In principle, data from these two instruments could be used to generate the same kind of cloud products as for AVHRR. For example, a new version of the NWC SAF PPS cloud processing package (to be used for compiling the CLARA-A3 TCDR) is already capable of processing SLSTR data. This information could thus be added to the other products generated from pure AVHRR-data and from the previously mentioned AVHRR-heritage sensors.

However, since Sentinel-3 is orbiting in a morning orbit (daytime equator crossing time at 10 am) the observations unfortunately tend to duplicate observations from other satellites (e.g., Metop satellites and EPS-SG satellites). Thus, the added value could be questioned (unless some of the other morning satellites fail). This question could be broadened to also include the information from other polar orbiting satellites, like Aqua and Terra with MODIS data and the Chinese polar orbiting FY-3D+ satellites with data from the MERSI-2 imager (the latter now also handled by the latest NWC SAF PPS package). Again, it is clear that the satellite orbits used for these satellites are practically identical to the morning and afternoon satellites of the NOAA and Metop series of satellites and their successors. Thus, the added value from these observations will probably only be marginal.

More important seems to be to build the dataset consistently on observations from one morning satellite and one afternoon satellite (i.e., the prime satellites as demonstrated by Stengel et al., 2017). A CDR based on only these prime satellites appears to be the most useful dataset from a climate monitoring perspective. Other satellites outside the primary AVHRR and AVHRR-heritage family could mainly serve as an important backup source of information if some of the prime satellites fail.

3.6.2 Cloud Properties TCDR ESA AATSR v3.0 + ICDR ESA SLSTR v3.1.1+v4.0

The ESA SLSTR v3.x ICDR directly exploits data from the Sentinel-3 platform. There have been examples shown of utilizing Sentinel-3 OLCI-like measurements (mainly using MERIS on ENVISAT) for cloud retrieval in conjunction with (A)ATSR or SLSTR (Carbajal Henken et al. 2014), but difficulties in cross-calibration and co-registration of the different instruments have meant these products have not shown improved performance over the (A)ATSR/SLSTR only algorithms. The availability of a well co-located and calibrated joint SLSTR-OLCI L1 product, could resurrect this approach to further improving cloud products derived from Sentinel-3 (and the preceding ENVISAT).

As discussed in section 3.4.2, the ORAC retrieval scheme can be, and has been, applied to a wide range of satellite visible-IR imaging radiometers. A particular instrument, of direct relevance to the Sentinel satellite program, is the Flexible Combined Imager (FCI) to fly on MeteoSat Third Generation/Sentinel-4. This instrument is essentially a replacement for the SEVIRI sensors on MSG, with capabilities similar to those provided by Himawari-AHI and GOES-ABI imagers (which ORAC has already been applied to).

References

Baum B.A. et al (2010), The impact of ice particle roughness on the scattering phase matrix, Journal of Quantitative Spectroscopy and Radiative Transfer, Volume 111, Issues 17–18, https://doi.org/10.1016/j.jqsrt.2010.07.008.

Baum B.A. et al (2014), Ice cloud single-scattering property models with the full phase matrix at wavelengths from 0.2 to 100µm, Journal of Quantitative Spectroscopy and Radiative Transfer, Volume 146, https://doi.org/10.1016/j.jqsrt.2014.02.029.

Brodzik, M. J. and J. S. Stewart. 2016: Near-Real-Time SSM/I-SSMIS EASE-Grid Daily Global Ice Concentration and Snow Extent, Version 5. Boulder, Colorado USA. NASA National Snow and Ice Data Center Distributed Active Archive Center. doi: https://doi.org/10.5067/3KB2JPLFPK3R.

Carbajal Henken, C.K., R. Lindstrot, R. Preusker and J. Fischer, 2014: FAME-C: cloud property retrieval using synergistic AATSR and MERIS observations. Atmos. Meas. Tech., 7, 3873–3890, doi:10.5194/amt-7-3873-2014.

CM SAF, 2022: Evaluation of the preliminary release of the FIDUCEO AVHRR GAC FCDR, SAF/CM/SMHI/REP/FDR, v. 1.0, (submitted to EUMETSAT).

Dee et al. (35 named authors), 2011: The ERA-Interim reanalysis: configuration and performance of the data assimilation system, Quart. J. Royal Met. Soc., 137 (656), 553 - 597. Online version: http://onlinelibrary.wiley.com/doi/10.1002/qj.828/abstract.

EUMETSAT, 2023 : PyGAC AVHRR FDR Release 1 – Product Users Guide, Document EUM/OPS/DOC/21/1269950, DOI : 10.15770/EUM_SEC_CLM_0060.

Stoffelen, A., 1998: Towards the true near-surface wind speed: Error modeling and calibration using triple collocation. J. Geophys. Res. 103C3, 7755-7766.

Dybbroe, A., A. Thoss and K.-G. Karlsson, 2005: NWCSAF AVHRR cloud detection and analysis using dynamic thresholds and radiative transfer modeling - Part I: Algorithm description, J. Appl. Meteor, 44, 39-54.

Eidenshink, J. and Faundeen, J., 1994: The 1 km AVHRR global land data set-first stages in implementation. Int. J. Rem. Sens., 15(17), 3443-3462.

EUMETSAT OSI SAF, 2015: Global sea ice concentration climate data records 1978-2015 (v1.2, 2015), [Online]. Norwegian and Danish Meteorological Institutes. doi: 10.15770/EUM_SAF_OSI_0001 and 10.15770/EUM_SAF_OSI_0005

Han, Q., W.B. Rossow and A.A. Lasis, 1994: Near-Global Survey of Effective Droplet Radii in Liquid Water Clouds using ISCCP Data, J. Climate, 7, 465-497, 1994.

Heidinger, A.K.; Foster, M.J.; Walther, A.; Zhao, Z., 2014: The Pathfinder Atmospheres Extended (PATMOS-x) AVHRR climate data set. Bull. Am. Meteorol. Soc., 95, 909–922.

Heidinger, A.K., W.C. Straka, C.C. Molling, J.T. Sullivan and X.Q. Wu, 2010: Deriving an inter-sensor consistent calibration for the AVHRR solar reflectance data record. Int. J. Rem. Sens., 31(24), 6493-6517.

Håkansson, N., Adok, C., Thoss, A., Scheirer, R., and Hörnquist, S., 2018: Neural network cloud top pressure and height for MODIS, Atmos. Meas. Tech., 11, 3177-3196, https://doi.org/10.5194/amt-11-3177-2018.

Karlsson, K.-G., Anttila, K., Trentmann, J., Stengel, M., Fokke Meirink, J., Devasthale, A., Hanschmann, T., Kothe, S., Jääskeläinen, E., Sedlar, J., Benas, N., van Zadelhoff, G.-J., Schlundt, C., Stein, D., Finkensieper, S., Håkansson, N., and Hollmann, R.: CLARA-A2: the second edition of the CM SAF cloud and radiation data record from 34 years of global AVHRR data, Atmos. Chem. Phys., 17, 5809-5828, https://doi.org/10.5194/acp-17-5809-2017, 2017.

Karlsson, K.-G. and Devasthale, A., 2018: Inter-Comparison and Evaluation of the Four Longest Satellite-Derived Cloud Climate Data Records: CLARA-A2, ESA Cloud CCI V3, ISCCP-HGM, and PATMOS-x, Rem. Sens., 10, 1567, doi:10.3390/rs10101567

Karlsson, K.-G., Håkansson, N., Mittaz, J. P. D., Hanschmann, T. and Devasthale, A., 2017: Impact of AVHRR Channel 3b Noise on Climate Data Records: Filtering Method Applied to the CM SAF CLARA-A2 Data Record, Rem. Sens., 9, 568, doi: 10.3390/rs9060568.

Karlsson, K.-G., E. Johansson and A. Devasthale, 2015: Advancing the uncertainty characterisation of cloud masking in passive satellite imagery: Probabilistic formulations for NOAA AVHRR data, Rem. Sens. Env. , 158, 126-139; doi:10.1016/j.rse.2014.10.028.

Karlsson, K.-G., Johansson, E., Håkansson, N., Sedlar, J. and Eliasson, S., 2020: Probabilistic cloud masking for the generation of CM SAF cloud climate data records from AVHRR and SEVIRI sensors, accepted for publication in Rem. Sens.

McGarragh, G.R.; Poulsen, C.A.; Thomas, G.E.; Povey, A.C.; Sus, O.; Stapelberg, S.; Schlundt, C.; Proud, S.; Christensen, M.W.; Stengel, M.; et al., 2018: The Community Cloud retrieval for CLimate (CC4CL). Part II: The optimal estimation approach. Atmos. Meas. Tech., 11, 3397–3431.

Moody, E.G., M.D. King, C.B. Schaaf, and S. Platnick, 2008: MODIS-Derived Spatially Complete Surface Albedo Products: Spatial and Temporal Pixel Distribution and Zonal Averages. J. Appl. Meteorol. Clim., 47, 2879–2894.

Nakajima, T., and M. D. King, 1990: Determination of the Optical Thickness and Effective Particle Radius of Clouds from Reflected Solar Radiation Measurements. Part 1: Theory. J. Atmos. Sci ., 47, 1878-1893.

Pavalonis, M.J., A.K. Heidinger and T. Uttal, 2005: Daytime Global Cloud Typing from AVHRR and VIIRS: Algorithm Description, Validation and Comparisons. J. Appl. Meteor., 44, 804-826. DOI:10.1175/JAM2236.1.

Pfreundschuh, S., Eriksson, P., Duncan, D., Rydberg, B., Håkansson, N., and Thoss, A., 2018: A neural network approach to estimating a posteriori distributions of Bayesian retrieval problems, Atmos. Meas. Tech., 11, 4627-4643, https://doi.org/10.5194/amt-11-4627-2018.

Poulsen, C.A, G. McGarragh, M. Stengal, 2018: ESA Cloud_cci Validation Report for MODIS multi-layer cloud. Report, accessed 19-Dec-2019: http://www.esa-cloud-cci.org/sites/default/files/upload/Cloud_cci_D4.1.2_RMLEV_v1.1.pdf

Schaaf, C. and Wang, Z., 2015: MCD43C1 MODIS/Terra+Aqua BRDF/AlbedoModel Parameters Daily L3 Global 0.05Deg CMG V006, NASA EOSDIS Land Processes DAAC, https://doi.org/10.5067/MODIS/MCD43C1.006.

Smith D. 2012: ENVISAT AATSR Instrument Performance – End of Mission Report. Report, accessed 02-Mar-2020: https://atsrsensors.org/pdf/ENVISAT%20AATSR%20Instrument%20Performance%20-%20End%20of%20Mission%20Report%20%20-%20Issue%201.0.pdf

Stengel, M., Stapelberg, S., Sus, O., Finkensieper, S., Würzler, B., Philipp, D., Hollmann, R., Poulsen, C., Christensen, M.; and McGarragh, G., 2020: Cloud_cci Advanced Very High Resolution Radiometer post meridiem (AVHRR-PM) dataset version 3: 35 year climatology of global cloud and radiation properties, Earth Syst. Sci. Data, 12, 41-60, https://doi.org/10.5194/essd-12-41-2020.

Stengel, M, C. Poulsen, R. Hollmann, N. Schneider, L. Gómez Chova, O. Sus, C. Schlundt, S. Testorp, 2018: ESA Cloud_cci Comprehensive Error Characterisation Report, Issue 4, Rev. 1. Accessed 19-Dec-2019: http://www.esa-cloud-cci.org/sites/default/files/upload/Cloud_cci_D1.3_CECR_v4.1.pdf

Stengel, M.; Stapelberg, S.; Sus, O.; Schlundt, C.; Poulsen, C.; Thomas, G.; Christensen, M.; Carbajal Henken, C.; Preusker, R.; Fischer, J.; et al., 2017: Cloud property datasets retrieved from AVHRR, MODIS, AATSR and MERIS in the framework of the Cloud_cci project. Earth Syst. Sci. Data, 9, 881–904.

Stephens, G.., 1978: Radiation profiles in extended water clouds. II: Parameterization schemes. J. Atmos. Sci., 35, 2123-2132.

Stoffelen , A., 1998: Towards the true near-surface wind speed: Error modeling and calibration using triple collocation. J. Geophys. Res. 103C3, 7755-7766.

Sun, B., Free, M., Yoo, H. L., Foster, M. J., Heidinger, A. and Karlsson, K.-G., 2015: Variability and Trends in U.S. Cloud Cover: ISCCP, PATMOS-x and CLARA-A1 Compared to Homegeneity-Adjusted Weather Observations, J. Clim., 28, 4373-4389, https://doi.org/10.1175/JCLI-D-14-00805.1.

Sus, O.; Stengel, M.; Stapelberg, S.; McGarragh, G.; Poulsen, C.; Povey, A.C.; Schlundt, C.; Thomas, G.; Christensen, M.; Proud, S.; et al., 2018: The Community Cloud retrieval for Climate (CC4CL). Part I: A framework applied to multiple satellite imaging sensors. Atmos. Meas. Tech., 11, 3373–3396.

USGS, 1996: Global 30 arc seconds topography database, GTOPO30, (http://edcdaac.usgs.gov/gtopo30/)

Wan, Z., Hook, S., Hulley, G., 2015: MYD11C3 MODIS/Aqua Land Surface Temperature/Emissivity Monthly L3 Global 0.05Deg CMG V006 [Data set]. NASA EOSDIS LP DAAC. doi: 10.5067/MODIS/MYD11C3.006

Winker, D. M., Vaughan, M. A., Omar, A., Hu, Y. and Powell, K. A., 2009: Overview of the CALIPSO mission and CALIOP data processing algorithms, J. Atmos. Oceanic. Technol., 26, 2310-2323, doi: 10.1175/2009JTECHA1281.1.

Related articles

• Page:
  CP CLARA-A2: Product User Manual (PUM) (Copernicus Knowledge Base)

  • ecv
  • clara
  • cloud_properties
  • owned-single-by-usmg
• Page:
  CP CLARA-A2: Algorithm Theoretical Basis Documents (ATBD) (Copernicus Knowledge Base)

  • ecv
  • clara
  • cloud_properties
  • owned-single-by-usmg
• Page:
  CP CLARA-A2: Validation report (Copernicus Knowledge Base)

  • ecv
  • clara
  • cloud_properties
  • owned-single-by-usmg