Contributors: G.E. Thomas (UKRI-STFC RAL Space)

Issued by: STFC RAL Space (UKRI-STFC) / Gareth Thomas

Date: 22/02/2023

Ref: C3S2_D312a_Lot1.3.2.2_202302_SQAD_CCI_CP_SRB_ERB_v1.1

Official reference number service contract: 2021/C3S2_312a_Lot1_DWD/SC1

Table of Contents

History of modifications

Version

Date

Description of modification

Chapters / Sections

V1.0

30/12/2022

Initial version

all

V1.1

22/02/2023

Updated to address reviewer's comments

all

List of datasets covered by this document

Deliverable ID

Product title

Product type (CDR, ICDR)

Version number

Delivery date

D3.3.17-v3.0

ECV Cloud Properties brokered from ESA´s Cloud_cci ATSR-2-AATSRv3 dataset

TCDR

V3.0

30/04/2020

D3.3.18-v3.x

ECV Cloud Properties derived from SLSTR

ICDR

V3.1

30/11/2020 - onward

D2.1.1-P1/2
D2.1.3-P1

ECV Cloud Properties derived from SLSTR

ICDR

V3.1.1
V4.0

30/04/2020





30/11/2020 - 30/09/2021

D3.3.8-v3.0

ECV Surface Radiation Budget brokered from ESA´s Cloud_cci ATSR-AATSRv3 dataset

TCDR

V3.0

31/05/2022 - onward

D3.3.7-v3.x

ECV Surface Radiation Budget derived from SLSTR

ICDR

V3.1

30/11/2020 - onward

D2.1.1-P1/2

D2.1.3-P1

ECV Surface Radiation Budget derived from SLSTR

ICDR

V3.1.1

V4.0

31/05/2022 - onward






D3.3.25-v3.0

ECV Earth Radiation Budget brokered from ESA´s Cloud_cci ATSR-AATSRv3 dataset

TCDR

V3.0

22/07/2020

D3.3.26-v3.x

ECV Earth Radiation Budget derived from SLSTR

ICDR

V3.1

30/11/2020 – onward

D2.1.1-P1/2

D2.1.3-P1

ECV Earth Radiation Budget derived from SLSTR

ICDR

V3.1.1

V4.0

31/05/2022 - onward

Related documents

Reference ID

Document

D1

Algorithm Theoretical Basis Document, v.6.2. ESA Cloud_cci.

https://climate.esa.int/media/documents/Cloud_Algorithm-Theoretical-Baseline-Document-ATBD_v6.2.pdf

Last accessed on 16/05/2023

D2

Algorithm Theoretical Basis Document: CC4Cl Broadband Radiative Flux Retrieval, v1.1. ESA Cloud_cci,

https://climate.esa.int/media/documents/Cloud_Algorithm-Theoretical-Baseline-Document-ATBD-CC4CL-TOAFLUX_v1.1.pdf

Last accessed on 16/05/2023

D3

Product User Guide, v5.1. ESA Cloud_cci,

https://climate.esa.int/media/documents/Cloud_Product-User-Guide-PUG_v5.1.pdf 

Last accessed on 16/05/2023

D4

Thomas, G. (2023) C3S Cloud Properties

Service: Algorithm Theoretical Basis Document. Copernicus Climate Change Service.

Document ref. C3S2_D312a_Lot1.2.1.3-v4.0_202301_ATBD_CCICloudProperties_v1.2

CP CCI ICDR: Algorithm Theoretical Basis Document

Last accessed on 16/05/2023

Acronyms

Acronym

Definition

(A)ATSR

(Advanced) Along-Track Scanning Radiometer

ATSR-1, -2

Along Track Scanning Radiometer

BRFR

Broadband Radiative Flux Retrieval

C3S

Copernicus Climate Change Service

CC4Cl

Community Cloud retrieval for Climate

CCI

(ESA) Climate Change Initiative

CDR

Climate Data Record

CDS

Climate Data Store

CEDA

Centre for Environmental Data Analysis

CKB

Copernicus Knowledge Base

CP

Cloud Properties

CUS

Copernicus User Service

ECMWF

European Centre for Medium Range Weather Forecasting

ECV

Essential Climate Variable

ERA

ECMWF ReAnalysis

ERB

Earth Radiation Budget

ESA

European Space Agency

FTP

File Transfer Protocol

HTTP

Hypertext Transfer Protocol

ICDR

Interim Climate Data Record

MODIS

Moderate Resolution Imaging Spectroradiometer

OPeNDAP

Open-source Project for a Network Data Access Protocol

ORAC

Optimal Retrieval of Aerosol and Cloud

RAL

Rutherford Appleton Laboratory

RTTOV

Radiative Transfer for TOVS (TIROS Observational Vertical Sounder)

SLSTR

Sea and Land Surface Temperature Radiometer

SOHO

Solar and Heliospheric Observatory

SORCE

SOlar Radiation and Climate Experiment

SRB

Surface Radiation Budget

STFC

Science and Technology Facilities Council

TCDR

Thematic Climate Data Record

TIROS

Television and Infra-Red Observation Satellite

UKRI

UK Research and Innovation

List of tables

List of figures

Figure 5-1: The schematic of Copernicus User Support (Copernicus User Support Handbook, V2.1)

General definitions

Table 1: Definition of processing levels

Processing level

Definition

Level-1b

The full-resolution geolocated radiometric measurements (for each view and each channel), rebinned onto a regular spatial grid.

Level-2 (L2)

Retrieved cloud variables at full input data resolution, thus with the same resolution and location as the sensor measurements (Level-1b).

Level-3C (L3C)

Cloud properties of Level-2 orbits of one single sensor combined (averaged) on a global spatial grid. Both daily and monthly products provided through C3S are Level-3C.

Table 2: Definition of various technical terms used in the document

Jargon

Definition

Brokered product

The C3S Climate Data Store (CDS) provides both data produced specifically for C3S and so-called brokered products. The latter are existing products produced under an independent programme or project which are made available through the CDS.

Climate Data Store (CDS)

The front-end and delivery mechanism for data made available through C3S.

OPeNDAP

“Open-Source Project for a Network Data Access Protocol” enables scientists to share data more easily over the internet. With OPeNDAP, one can access data using an OPeNDAP URL of any database server that supports OPeNDAP. It can be done via command-line, Internet browser, or a custom UI. (from https://www.earthdata.nasa.gov/engage/open-data-services-and-software/api/opendap)

JASMIN

JASMIN is a data analysis facility, which provides storage and compute facilities to enable data-intensive environmental science.

This is where the data processing takes place.

Merged version

The two platforms (Sentinel-3A/B) fly in identical, interleaved sun-synchronous orbits, such that the two SLSTR instruments provide nearly global coverage twice daily (with one day and one night overpass).

Two products are provided: The data records of each satellite (S3A and S3B) individually as well as a merged product of both for the period in which both are available (10/2018 – 06/2022). (see D4, Section 1)

TCDR

It is a consistently-processed time series of a geophysical variable of sufficient length and quality.

ICDR

An Interim Climate Data Record (ICDR) denotes an extension of TCDR, processed with a processing system as consistent as possible to the generation of TCDR.

CDR

A Climate Data Record (CDR) is defined as a time series of measurements with sufficient length, consistency, and continuity to determine climate variability and change.

Scope of the document

This document describes the systems used for delivering the satellite-based estimates of Cloud Properties (CP), Surface Radiation Budget (SRB), and Earth Radiation Budget (ERB), brokered from ESA´s Cloud_cci ATSR-AATSRv3 dataset or derived from the Sea and Land Surface Temperature Radiometer (SLSTR) specifically for the Copernicus Climate Change Service (C3S). The CP, SRB, and ERB products are produced by the Optimal Estimation of Aerosol and Cloud (ORAC) retrieval scheme, operating as the Community Cloud for Climate (CC4Cl) processor, and provided to the Climate Data Store (CDS) by the C3S. Interfaces to external data, the data processing infrastructure, and the interface to the CDS are outlined. Internal procedures for managing and communicating system maintenance and for user support are described.

Executive summary

All ORAC datasets – brokered (A)ATSR cloud and radiation CDRs and SLSTR cloud and radiation ICDRs – are made available to the CDS using a common architecture. Access is via HTTP access to the CDS-C3S cloud workspace hosted on the JASMIN compute infrastructure provided by the Centre for Environmental Data Analysis (CEDA), or through CEDA’s long-term data archive with FTP, HTTP, or OPeNDAP interfaces available. As a national facility, JASMIN outages are kept to a minimum and, if possible, will be communicated to C3S in advance. We provide user support for queries relating to our products when forwarded to us by C3S (see Section 5 User Support).

1. System overview

1.1 System elements and interfaces

The Cloud_cci ATSR-AATSRv3 dataset has been produced and archived outside of this contract by ESA, whereas the SLSTR extension Interim Climate Data Record (ICDR) is produced specifically for C3S, using the same processing infrastructure as the original Cloud_cci TCDR. Data was produced using the CC4CL cloud processor, with the Optimal Retrieval of Aerosol and Cloud (ORAC) algorithm producing cloud properties, which are then fed into the Broadband Radiative Flux Retrieval (BRFR) module to produce the Surface Radiation Budget (SRB) and Earth Radiation Budget (ERB) products. The CC4CL processor consists of a 4 or 5-stage processing chain, depending if only CP products or ERB/SRB products are required, to go from level-1 satellite top-of-atmosphere radiances to level-3 gridded  data:

  1. Pre-processing: Reads satellite level-1 data, as well as ancillary data including meteorology data (ECMWF ERA-Interim in the case of the v3 Cloud_cci product and ERA-5 in the case of the SLSTR ICDR) and surface reflectance (MODIS MCD43A1 BRDF1 product). Clear-sky radiative transfer calculations are done, using RTTOV (version 12), data are co-located onto the level-1 data grid and all inputs required by the ORAC retrieval code are written to disk.
  2. Cloud-retrieval: The cloud retrieval is run twice, one assuming water cloud droplets, one assuming ice.
  3. Post-processing: This step combines the ice and water cloud retrievals into a unified product, based on which assumption best fit the observations, and the retrieved cloud-top-temperature. This step produces the level-2 cloud product.
  4. Flux calculation2: The BRFR module reads the level-2 cloud product, temperature, humidity, ozone and surface reflectance data produced by the pre-processing step, (A)ATSR aerosol_cci level-2 data (also produced by the ORAC processor) and solar irradiance data provided by the Solar and Heliospheric Observatory (SOHO) (https://sohowww.nascom.nasa.gov/) and SOlar Radiation and Climate Experiment (SORCE) (https://lasp.colorado.edu/home/sorce/) satellite records. This step produces the level-2 radiative flux product.
  5. Level-3 compositing: Level-2 cloud properties and radiative fluxes are averaged onto the regular lat-lon grid used for the level-3 products. This step produces the level-3 monthly CP, SRB and ERB data provided to C3S.

All steps of this processing are coded in Fortran 2003 and are version controlled using Git. All code, with the exception of the level-3 processor, are publicly accessible, under the GNU General Public License v3, via GitHub (https://github.com/ORAC-CC/orac).

Details of the cloud retrieval algorithm can be found in the ATBD [D1], while the BRFR module has its own ATBD [D2]. Details of the Cloud_cci product in general can be found in the product user guide [D3].

Processing is performed using the Lotus cluster, which provides a large-scale computational resource (approx. 12,000 CPU cores) to the JASMIN super-data-cluster. Use of JASMIN for processing provides the benefit that all required level-1 satellite and ancillary data are available from on-line disk storage. The processing system is run via Python and shell scripts, interfacing to a Slurm3 job-manager. Reformatting of the Cloud_cci level-3 products to meet C3S requirements is performed using run-time IDL code.

1 The product used is MCD43A1 V006. See the MODIS user guide at https://www.umb.edu/spectralmass/terra_aqua_modis/v006 (accessed 30-Jan-2023).

2 Only for SRB and ERB

3 Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system: https://slurm.schedmd.com/overview.html

1.1.1 Interfaces to external data

Brokered CDR products

The Cloud_cci TCDR is a brokered product, for which processing has been completed. The brokered data is static and self-contained – no further production of version-3 Cloud_cci data will be undertaken.

Interim CDR products

All required input data needed to produce ICDR data from SLSTR is stored on-line on the JASMIN system, including ERA data (which is provided by CEDA) and the auxiliary datasets of MODIS MCD43A1.

1.1.2 Delivery of the data to C3S

Access to Cloud_cci Cloud Properties, Surface Radiation Budget, and Earth Radiation Budget datasets is provided using the same mechanism as used for the CCI project itself, and is replicated for SLSTR ICDR products. Data are provided through an HTTP access to CEDA group-workspace storage until data is transferred into the CEDA archive, after which FTP, HTTP, and OPeNDAP are all available.

The storage (both group-workspace and archive) used for storing the brokered Cloud_cci data and SLSTR ICDR data, as well as the HTTP, FTP, and OPeNDAP interfaces are provided and maintained by CEDA, as part of the UK national computational infrastructure. See section 1.2 for further details of this system.

1.2 Hardware, supercomputers and cloud computing

1.2.1 Computing infrastructure

The UK JASMIN "super-data-cluster", operated by CEDA, provides all the computing infrastructure for production and delivery of ORAC products. JASMIN hosts the UK archive of both (A)ATSR and SLSTR level 1b data, as well as providing on-line access to ECMWF ERA-5 data. Through support from the UK Space Agency, the system also provides approximately 12,000 CPU cores for data production and 250 Tb of storage for data production and provision, dedicated solely for C3S_312b_Lot1.

1.2.2 Back-up

(A)ATSR and SLSTR products are archived and served by the Centre for Environmental Data Analysis (CEDA) which holds multiple tape backups of its archive. Data stored on group-workspace storage are not automatically backed up, but a back-up can be made through a user-accessible elastic tape storage system.

1.2.3 Data access and delivery

The CEDA archive provides HTTP, FTP and OpenDAP access to datasets. Access to data stored on the dedicated C3S_312b_Lot1 group-workspace will be through HTTP only. CEDA provides a target minimum availability of both the archive and workspace servers of 95%, with data transfer rates greater than 10 Mbit/s.

2. Upgrade cycle implementation procedure

The data record provided by the Cloud_cci ATSR-2/AATSR dataset will be extended by the production of an ICDR from the Sea and Land Surface Temperature Radiometer (SLSTR) instrument, flying on board the Sentinel-3 satellites from mid-2016. This record will be updated to include the most recent data on a 6-monthly basis.

3. Procedures for reprocessing CDRs

Within the scope of this project, no reprocessing of the ESA Cloud_cci ATSR-AATSRv3, or SLSTR, cloud properties dataset is foreseen. Therefore n/a.

4. System maintenance and system failures

In general, any issues with IT infrastructure that either delay delivery of data or impede access to data via the CDS will be documented in the Quarterly Report.

4.1 Interfaces to external data

See section 1 above.

4.2 Production of data sets

No data production is required to deliver Cloud_cci (A)ATSR data, as it is an existing, brokered dataset. Production of SLSTR ICDR data makes use of the Lotus cluster, which forms part of the JASMIN infrastructure.

4.3 Interface to the CDS

This section discusses maintenance and outages of the data server hosting the files for access through the CDS.

4.3.1 Planned maintenance

CEDA performs regular maintenance and upgrades of its systems, and these works are undertaken in such a way to minimize impact to users of the system. Scheduled updates and downtime are advertised on the CEDA website and through an RSS feed (http://www.ceda.ac.uk/blog/feeds/rss/).
CEDA also takes the needs of its customers and users into account when planning downtime. These notifications will cover interruptions to input data, processing infrastructure and access to data for C3S.

4.3.2 Unplanned outages

CEDA has a dedicated technical team to deal with unplanned outages, and their helpdesk keeps users informed through emails and news-feed updates. This information will be forwarded on to C3S as required.

5. User support

5.1 Enquiries

The user (customer) has the possibility to browse and search in the Copernicus Knowledge Base (CKB) (CAMS and C3S Knowledge Base (CKB)) or the Copernicus Climate Data Store desk (https://cds.climate.copernicus.eu/cdsapp#!/usersupport) (Figure 5-1). This is known as the level-0 process of User Support.

Figure 5-1: The schematic of Copernicus User Support (Copernicus User Support Handbook, V2.1)


Contact and User Support process on JIRA service desk

Once a request is sent, the Copernicus User Support (CUS) Service team at ECMWF will handle the requests within 8 hours (level 1).

For any scientific and special enquiries that cannot be answered by the CUS team at ECMWF or addressed to the Knowledge Base, the request will be forwarded to the Copernicus User Support Specialists (level-2).

Enquiries forwarded to the Copernicus User Support Specialist team will be acknowledged within 3 working days (target 100%) and a notification sent to the user. In case of specific scientific issues, the enquiries will be channelled to the ECV and data specialist of the C3S2_312a_Lot1 project and should be resolved within 3 working weeks (target 85%). In each quarter, we aim for User Support satisfaction scoring 3 in 90% of all voluntary based feedbacks by users, with 1 (very unsatisfied) to 5 (very satisfied). We will also list the number of tickets in the Quarterly Report.

This document has been produced in the context of the Copernicus Climate Change Service (C3S).

The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts, operator of C3S on behalf on the European Union (Contribution Agreement signed on 22/07/2021). All information in this document is provided “as is” and no guarantee of warranty is given that the information is fit for any particular purpose.

The users thereof use the information at their sole risk and liability. For the avoidance of all doubt, the European Commission and the European Centre for Medium-Range Weather Forecasts have no liability in respect of this document, which is merely representing the author’s view.

Related articles