Contributors: C. Reimer (EODC GmbH), R. Kidd (EODC GmbH), A. Dostalova (EODC GmbH), R. van der Schalie (Vandersat/ Planet labs), C. Paulik (Vandersat/ Planet labs), W. Preimesberger (TU Wien)

Issued by: EODC GmbH/Alena Dostalova

Date: 25/09/2023

Ref: C3S2_312a_Lot4.WP3-SQAD-SM-v1_202301_SM_System_Quality_Assurance_i1.1

Official reference number service contract: 2021/C3S2_312a_Lot4_EODC/SC1

   

Table of Contents

History of modifications

Version

Date

Description of modification

Chapters / Sections

i0.1

17/01/2023

Created document based on SQAD v3.0 (D1.SM.1-v3.0) Revision of all datasets covered by the document, revision of all deliverable IDs for related documents, removal of references not called in the document. Review of all sections, updated to the current status.

All

i1.0

26/01/2023

Document finalized

All

i1.1

06/09/2023

Document amended in response to independent review

All

List of datasets covered by this document

Deliverable ID

Product title

Product type (CDR, ICDR)

C3S version number

Public version number

Delivery date

WP2-FDDP-SM-CDR-v4

Surface Soil Moisture (Passive) Daily

CDR

v4.0

v202212

31/12/2022

WP2-FDDP-SM-CDR-v4

Surface Soil Moisture (Passive) Dekadal

CDR

v4.0

v202212

31/12/2022

WP2-FDDP-SM-CDR-v4

Surface Soil Moisture (Passive) Monthly

CDR

v4.0

v202212

31/12/2022

WP2-FDDP-SM-CDR-v4

Surface Soil Moisture (Active) Daily

CDR

v4.0

v202212

31/12/2022

WP2-FDDP-SM-CDR-v4

Surface Soil Moisture (Active) Dekadal

CDR

v4.0

v202212

31/12/2022

WP2-FDDP-SM-CDR-v4

Surface Soil Moisture (Active) Monthly

CDR

v4.0

v202212

31/12/2022

WP2-FDDP-SM-CDR-v4

Surface Soil Moisture (Combined) Daily

CDR

v4.0

v202212

31/12/2022

WP2-FDDP-SM-CDR-v4

Surface Soil Moisture (Combined) Dekadal

CDR

v4.0

v202212

31/12/2022

WP2-FDDP-SM-CDR-v4

Surface Soil Moisture (Combined) Monthly

CDR

v4.0

v202212

31/12/2022

Related documents

Reference ID

Document

D1

Preimesberger, W., et al (2022). C3S Soil Moisture Version v202212: Product Quality Assurance Document. Document ref.: C3S2_312a_Lot4.WP1-PDDP-SM-v1_202206_SM_PQAD-v4_i1.1

D2

Preimesberger, W., et al (2023). C3S Soil Moisture Version v202212: Algorithm Theoretical Baseline Document. Document ref.: C3S2_312a_Lot4.WP2-FDDP-SM-v1_202212_SM_ATBD-v4_i1.1

D3

Kidd, R., et al (2019). C3S Land Hydrology Cryosphere: Updated KPI's. Document ref.: C3S_312b_Lot4.D0.S.8_Updated_Key_Performance_Indicators_v1.2

Acronyms

Acronym

Definition

AMI-WS

Active Microwave Instrument Wide Swath

AMSR2

Advanced Microwave Scanning Radiometer 2

AMSR-E

Advanced Microwave Scanning Radiometer-Earth Observing System

ASCAT

Advanced Scatterometer (Metop)

ATBD

Algorithm Theoretical Basis Document

BUFR

Binary Universal Form for the Representation of meteorological data

C3S

Copernicus Climate Change Service

C3S ECV SM PS

C3S Soil Moisture production system

CAMS

Copernicus Atmosphere Monitoring Service

CCI

Climate Change Initiative

CDF

Cumulative Distribution Function

CDR

Climate Data Record

CDS

Climate Data Store

CPF

C3S Processing Framework

CPU

Central Processing Unit

CUS

Copernicus User Support

ECMWF

European Center for Medium-range Weather Forecasts

ECV

Essential Climate Variable

EODC

Earth Observation Data Centre

ERS

European Remote Sensing Satellite (ESA)

ESA

European Space Agency

EUMETSAT

European Organisation for the Exploitation of Meteorological Satellites

GPM

Global Precipitation Measurement

HPC

High Performance Computing

H-SAF

EUMETSAT Satellite Application Facility on Support to Operational Hydrology and Water Management

ICDR

Intermediate CDR

KPI

Key Performance Indicators

LPRM

Land Parameter Retrieval model

METOP

Meteorological Operational Satellite (EUMETSAT)

NASA

National Aeronautics and Space Administration

NetCDF

Network Common Data Form

NRT

Near Real Time

PQAD

Product Quality Assurance Document

PVIR

Product Validation and Intercomparison Report

R&D

Research and Development

RAM

Random Access Memory

SIDP

Science Integration & Software Development Platform

SM

Soil Moisture

SMAP

Soil Moisture Active Passive

SMM/I

Special Sensor Microwave Imager

SMMR

Scanning Multichannel Microwave Radiometer

SMOS

Soil Moisture and Ocean Salinity

SQAD

System Quality Assurance Document

SSH

Secure Shell

TMI

TRMM Microwave Imager

TRMM

Tropical Rainfall Measuring Mission

VSC-5

Vienna Scientific Cluster 5

General definitions

Active (soil moisture) retrieval: the process of modelling soil moisture from radar (scatterometer and synthetic aperture radar) measurements. The measurand of active microwave remote sensing systems is called "backscatter".

Backscatter is the measurand of "active" microwave remote sensing systems (radar). As the energy pulses emitted by the radar hit the surface, a scattering effect occurs and part of the energy is reflected back. The received energy is called "backscatter", with rough surfaces producing stronger signals than smooth surfaces. It comprises reflections from the soil surface layer ("surface scatter"), vegetation ("volume scatter") and interactions of the two. Under very dry soil conditions, structural features in deeper soil layers can act as volume scatterers ("subsurface scattering").

Climate Data Record (CDR): Archive of the C3S soil moisture product ranging from 1978 to the date indicated in the product version name.

Intermediate Climate Data Record (ICDR): Consistent temporal extension of the given CDR record. It is produced every dekad (10 days) and published on CDS on a regular basis.

Level 2 pre-processed (L2P): this is a designation of satellite data processing level. "Level 2" means geophysical variables derived from Level 1 source data on the same grid (typically the satellite swath projection). "Pre-processed" means ancillary data and metadata added following GHRSST Data Specification.

Passive (soil moisture) retrieval: the process of modelling soil moisture from radiometer measurements. The measurand of passive microwave remote sensing is called "brightness temperature"). The retrieval model in the context of Copernicus Climate Change Service (C3S) soil moisture is generally the Land Parameter Retrieval Model.

Radiometer: Spaceborne radiometers are satellite-carried sensors that measure energy in the microwave domain emitted by the Earth. The amount of radiation emitted by an object in the microwave domain (~1-20 GHz). The observed quantity is called "brightness temperature" and depends on kinetic temperature of an object and its emissivity. Due to the high emissivity of water compared to dry matter, radiometer measurements of Earth's surface contain information in the water content in the observed area.

Scatterometer: Spaceborne scatterometers are satellite-carried sensors that use microwave radars to measure the reflection or scattering effect produced by scanning a large area on the surface of the Earth. The initially submitted pulses of energy are reflected by the Earth's surface depending on its geometrical and geophysical properties in the target area. The received energy is called "backscatter". Soil moisture retrieval relies on the fact that wet soils have a higher reflectivity (and therefore backscatter) than dry soils due to the high dielectric constant of liquid water compared to dry matter


Scope of the document

The System Quality Assurance Document outlines the architecture of the Copernicus Climate Change Service (C3S) Soil Moisture Production System with a detailed description of the involved system elements and their dependencies. The document aims to provide recipes to mitigate and handle possible scenarios affecting the operational service provided by the system.

Executive summary

The C3S Soil Moisture (SM) production system (C3S ECV SM PS) provides an operational soil moisture service, generating soil moisture climate data sets for a wide variety of users within the climate change community. Two teams support and maintain the system. The science team directs and implements the scientific evolution of the service and provides dedicated scientific support to the users. The Operations Team is responsible for the Technical Platform and the Operational Framework. The responsibilities of the Operations Team, with respect to system maintenance and system failures, are summarized in a list of actions to be triggered in case they are required.

This document details the integral components of the C3S ECV SM production system and their functions and interfaces (Section 1). The implementation procedure of the upgrade cycle is given in Section 2 while Section 3 describes the procedures for reprocessing of Climate Data Records (CDRs). Section 4 details procedures that are applied in case of system maintenance or failures and Section 5 provides information on the User support.

1. System overview

The C3S Soil Moisture (SM) production system (C3S ECV SM PS) is responsible for the operational provisioning of the soil moisture Essential Climate Variable (ECV) for the Copernicus Climate Change Service. The mission of the system is the timely production of a consistent, long-term, multi-mission, global soil moisture data set complying with the defined key performance indicators (KPI’s) [D3]. The intention of the system concept was to create a semi‑automated, intelligible, reproducible and cost‑effective system to foster the interoperability of the individual system elements and teams. Integral components of the C3S ECV SM PS are:

  • the Organizational Framework consisting of
    • the Science and
    • the Operations team
  • the Technical Platform and the Operations Framework
  • the C3S Processing Framework (CPF), and
  • Data Management and Streams

These components, their functions and interfaces within the C3S ECV SM production system will be discussed in the following.

1.1. Organizational Framework

Two teams have been elected within the Organizational Framework in order to fulfil the mission objective of the system. At first, the Science team is in charge of the physical reliability and validity of the soil moisture ECV product generated by the system. As a consequence, the function of the Science Team is to develop and maintain scientific algorithms capable to produce a long-term consistent, multi‑mission global soil moisture product in accordance with the defined data key performance indicators [D3]. Algorithms provided by the Science Team are source code packages to be integrated in the C3S Processing Framework (CPF). These source code packages are exchanged with the C3S ECV SM PS by utilising a well-defined interface realised by shared code repository (see Section 1.2). In addition, the Science Team is responsible to perform regular validation activities of the various SM ECV products released to the Copernicus Climate Change Service through the system.

The Operations Team, on the other hand, is in charge of managing the system and its individual components and takes control of the regular production of the soil moisture ECV. With respect to that, the Operations Team performs the orchestration, provisioning and configuration of the technical platform as well as the setup and maintenance of the Operations Framework (see Section 1.2). Members of the Operations Team exclusively have access to the Technical Platform and Operations Framework. Access by individuals outside of the Operations Team to these components of the system is not foreseen, in order to reduce and mitigate unpredictable system failures. Inputs provided by the Science Team are picked up by the Operations Team and are revised in an iterative and cooperative way before those are integrated in the operational C3S processing framework.

1.2. Technical Platform and Operations Framework

The Technical Platform and Operations Framework is composed of a set of hardware and software components interacting with each other. In the following a detailed description of the individual components will be given.

1.2.1. Technical Platform

The Technical Platform is implemented at the Earth Observation Data Centre (EODC) by providing three major hardware components fully embedded in the C3S ECV SM CPS. These hardware components are:

  • the EODC Cloud platform,
  • EO Storage, and
  • the Vienna Scientific Cluster 5 (VSC‑5)1

EO Storage is the centrepiece of the Technical Platform, accessible from both compute environments, EODC Cloud and VSC‑5, are dedicated to run the different production cycles of the soil moisture ECV. EO Storage is a multi-tier storage system composed of three tiers. The first tier of the storage system is a disk storage cluster (tier1) specifically design for high performance computing (HPC). The second storage tier is a slow on-demand tape storage pool (tier2) accompanied by the third tier a tape library (tier3). At the current stage of the C3S ECV SM CPS only tier1 and tier3 are considered (see Section 1.4). The disk storage (tier1) is foreseen to hold the entire archive of data required for the production of the latest version of the soil moisture ECV (Climate Data Record: CDR and Intermediate Climate Data Record: ICDR). Outdated, older versions of the soil moisture ECV (CDR and ICDR) with their corresponding input and auxiliary data are preserved in the tape library. 

During the conceptual phase of the system it was recognised that two distinct processing demands exist, related to the production of the soil moisture CDR and ICDR. It was decided to run the different production cycles, of the CDR and ICDR, in diverse compute environments fulfilling the processing demands and optimising the processing costs (see Figure 1). As a result, the production of the soil moisture CDR, with the demand of temporal, large‑scale compute resources, is carried out on the VSC-5. The benefit of using the VSC-5 is the scalability of the processing job, being able to parallelise individual tasks on thousands of Central Processing Units (CPUs). VSC-5 is a slurm2 based compute cluster for batch processing jobs. Jobs are submitted to a job queue and executed in sequence as soon as compute resources are available. Non-time critical processing jobs, such as the production of the CDR, are a perfect application for such large-scale compute clusters by minimising compute costs.

Complementary, the ICDR generation, with the demand of highly available, small‑scale compute resources, is performed on the EODC Cloud platform. This platform is devised as a virtualised, high‑availability compute cluster operated as a private cloud infrastructure based on OpenStack3. The C3S ECV SM PS is incorporated in OpenStack as a particular tenant to guarantee a certain level of security and encapsulate the system from other cloud users and applications. Within the OpenStack tenant a small-scale compute cluster is setup for the operational production of the ICDR. The current setup of the cluster comprises 4 nodes: 1 Gateway node (2 vCPUs, 2 GB RAM), 2 worker nodes (32 vCPUs, 96 and 32 GB RAM) and 1 Level 2 SM processor node (4 vCPUs, 4 GB RAM). The EODC Cloud allows to elastically scale compute resources on demand so that each of these nodes can be replicated to function as a test facility. This cluster is complemented by additional compute instances providing basic services to the C3S ECV SM PS such as a sftp server for data exchange and a web server for data provision to the CDS.

Figure 1: Technical Platform of the C3S ECV SM PS

1 https://vsc.ac.at//systems/vsc-5/ [The URL last viewed on 6th September 2023]

2 https://slurm.schedmd.com/ [the URL resource last viewed on 6th September 2023]

3 https://www.openstack.org/ [the URL resource last viewed on 6th September 2023]

1.2.2. Operations Framework

The Operations Framework is implemented as services and workflows to manage the timely production and delivery of the soil moisture ECV, integrating and building upon the Technical Platform. The centrepiece of the Operations Framework is a source code library base on Git4 and its web‑based repository manager GitLab5, holding all code and configurations fragments needed to run the operational service. Access to the GitLab code repository is restricted to members of the C3S soil moisture project consortium. Specific version of code packages can be cloned or downloaded to the Technical Platform. Furthermore, GitLab serves as the documentation platform of all packages by making use of the integrated WIKI platform.

System operations are done by members of the Operations Team. Team members have unrestricted access to the Technical Platform and to the Operations Framework as a single Technical User. This Technical User, with username c3s of the group c3s, handles the entire operational service of the C3S ECV SM PS. In order to separate the Level 2 soil moisture input data processing from the actual C3S ECV SM production a second user, with username c3s_lprm, is available. This user is responsible for the Near Real Time (NRT) passive input stream, that is the recurrent input data used in the ICDR production. Secure access to the Technical Platform and Operations Framework is exclusively given via Secure Shell (SSH) using private-key cryptography for authentication. At the moment it is not foreseen to add additional Technical Users.

4 https://git-scm.com/ [the URL resource last viewed on 6th September 2023]

5 https://about.gitlab.com/ [the URL resource last viewed on 6th September 2023]

1.3. C3S Processing Framework

The C3S Processing Framework (CPF) for soil moisture is based upon the scientific base of the European Space Agency (ESA) Climate Change Initiative (CCI) Soil Moisture project6. This means that the CPF uses the CCI processor of a specific version for merging of active and passive soil moisture products into a merged active, merged passive and combined soil moisture product. Pre‑ and post‑processing needed to get the input data streams into the correct format to be handled by the CCI processor, or to re‑format the data to the C3S specifications, is added around the CCI processor core. Figure 2 shows a high-level overview of the complete CPF while Figure 3 provides a more detailed look at the components of the framework.

6 https://esa-soilmoisture-cci.org/ [the URL resource last viewed on 6th September 2023]


Figure 2: Simplified overview of the C3S Processing Framework, structured into high level components.

Figure 3: Overview of the elements and interfaces of the C3S Processing Framework.

1.3.1. Interface to CCI processor

The close link to the Research and Development (R&D) activities in the CCI project is important for bringing new scientific knowledge into C3S. Input and output formats of the CCI processor will stay the same between versions which enables us to replace the CCI processor block (see Figure 2) in the processing framework without the need for large adaption work.

The CCI processor was adapted to run in different modes. In the “reprocessing” mode, default mode, the CCI processor computes processing parameters and produces the most consistent soil moisture CDR based on all available input datasets (historic and current missions). The “NRT” mode of the CCI processor uses these processing parameters to generate the ICDR, taking into account the latest available input data from all current missions, with less computational effort compared to the “reprocessing” mode. Processing parameters are generated during each reprocessing cycle. The parameters are Cumulative Distribution Function (CDF) scaling parameters used for scaling the different datasets into the same data space and an error characterization used for weighing different observations. These parameters are stored for each sensor combination and used in “NRT” mode of the processor. Figure 4 shows the scaling parameters and characterized errors for the production of the ICDR in “NRT” mode using the Advanced Scatterometer (ASCAT), Advanced Microwave Scanning Radiometer 2 (AMSR2) Soil Moisture and Ocean Salinity (SMOS), Soil Moisture Active Passive (SMAP) and Global Precipitation Measurement (GPM) sensors. More details about the algorithm and the used parameters can be found in the Algorithm Theoretical Baseline Document (ATBD) [D2].

Figure 4: Overview of the CCI processor in NRT mode

1.3.2. LPRM within C3S Processing Framework

The Land Parameter Retrieval Model (LPRM) is an integral part of the CPF, responsible for the timely production of Level 2 soil moisture products from passive microwave missions. Passive soil moisture products ingested in the C3S ECV SM CPS are produced by VanderSat using the processing facilities provided by the EODC through the C3S project. The historic data from the sensors Scanning Multichannel Microwave Radiometer (SMMR), Special Sensor Microwave Imager (SSM/I), Tropical Rainfall Measuring Mission (TRMM) Microwave Imager (TMI), Advanced Microwave Scanning Radiometer-Earth Observing System (AMSR-E), FengYun (FY) -3B/C/D and WindSat are reprocessed when algorithm updates in LPRM are developed. Data from AMSR2, SMOS, SMAP and GPM are processed in NRT mode.

1.3.3. Post processing

The CCI processor produces daily NetCDF files in both processing modes. Consequently, the C3S post processing block performs the dekadal and monthly mean calculation as well as data provisioning to the CDS.

1.3.4. ICDR: Workflow scheduling and monitoring

The production of the ICDR is a time critical task, because the result of each processing run has to be publicly available after a short latency. In order to meet this requirement the workflow, scheduling and monitoring tool Apache Airflow7 is used for the monitoring of the NRT input data streams on a daily basis. After ensuring that all NRT data streams are available for the last dekade, a member of the operations team triggers the automated processing for each currently active ICDR version on the worker nodes. Results and processing times of individual tasks and subtasks are monitored, and the final files are checked before their provision to the CDS.

7 https://airflow.incubator.apache.org/index.html [the URL resource last viewed on 6th September 2023]

1.4. Data Management and Streams

Data used in the C3S ECV SM PS can be discriminated into historic and NRT inputs. Historic data are soil moisture products from already decommissioned satellite missions generated in one processing cycle. On the other hand, NRT data are soil moisture products extended and processed on a regular basis as soon as new observations from current missions are available. Major characteristics of both historic and NRT input streams are summarized in Table 1; more information can be found in the ATBD [D2].

The entire data archive is stored and maintained on the EO Storage system operated by EODC. Raw data (Level 1 and 2) as disseminated by the dataset providers are stored besides the different versions of the various processing cycles of the C3S ECV SM PS. As of January 2023, the data archive comprising all C3S relevant datasets is about 10.3 TB currently including four versions of the CDR and ICDR (v201712, v201812, v201912 and v202012). This data archive is available on the disk storage cluster (tier1) of EO Storage and is complemented by regular backups of the data using the third tier the tape library.

For each processing cycle of the CDR and ICDR, the generated data will be placed in separate directories to assure data integrity of the different versions. Furthermore, data produced by test facilities of the system will be completely separated from the operational data stream for the same reason. The latest version of a soil moisture CDR and ICDR will reside on disk storage (tier1) as long as the data stream is operational. Outdated, older versions of the soil moisture ECV with its corresponding processing parameters are archived in the tape library (tier3).

Movement and deletion of data can only be done by the Operations Team and is closely coordinated with the Science Team.

NRT data streams required for the operational production of the ICDR are regularly monitored by the Operations Team by utilizing workflow routines implemented in Apache Airflow. These checks are executed beforef each ICDR production run raising an error if a data stream is broken.

Table 1: Major characteristics of input streams. NRT data streams are highlighted in green.


Sensor

Platform

Product

Product source

Time period used

Channel used for Soil Moisture

Spatial coverage

Passive microwave products

SMMR

Nimbus 7

LPRM (VanderSat)

NASA, Tape derived

11/1978 – 8/1987

6.6 GHz

Global

SSM/I

DMSP

LPRM (VanderSat)

NASA EarthData, XCAL calibrated with GPM

09/1987 – 2/2002

19.3 GHz

Global

TMI

TRMM

LPRM (VanderSat)

NASA EarthData, XCAL calibrated with GPM

01/1998 – 12/2013

10.7 GHz

N 40o to S 40o

AMSR-E

Aqua

LPRM (VanderSat)

JAXA, G-portal

07/2002 – 10/2011

6.9/10.7 GHz

Global

AMSR2

GCOM-W1

LPRM (VanderSat)

JAXA, G-portal

05/2012 – Present

6.925/10.65 GHz

Global

WindSat

Coriolis

LPRM (VanderSat)

Bespoke order

10/2007 – 7/2012

6.8/10.7 GHz

Global

MIRAS

SMOS

LPRM (VanderSat)

CATDS

01/2010 – Present

1.4 GHz

Global

SMAP

SMAP

LPRM (VanderSat)

NASA EarthData

04/2015- Present

1.4GHz

Global

GMI

GPM

LPRM (VanderSat)

NASA EarthData, XCAL calibrated with GPM

03/2014 – Present

10.7 GHz

N 70o to S 70o

MWRI

FY-3B

LPRM (VanderSat)

nsmc

06/2011 – 08/2019

10.7 GHz

Global

MWRI

FY-3C

LPRM (VanderSat)

nsmc

09/2013 – 02/2020

10.7 GHz

Global

MWRI

FY-3D

LPRM (VanderSat)

nsmc

07/2012 – 12/2021

10.7 GHz

Global

Active microwave products

AMI-WS

ERS1/2

SSM Product (TU WIEN, 2013)

TU WIEN Change Detection

7/1991 – 12/2006

5.3 GHz

Global

AMI-WS

ERS2

SSM Product (Crapolicchio et al., 2016)

TU WIEN Change Detection

5/1997 – 2/2007

5.3 GHz

Global

ASCAT

Metop-A

H 119 (H-SAF 2018a,b)

TU WIEN Change Detection

1/2007 – 11/2021

5.3 GHz

Global

ASCAT

Metop-B

H 119 (H-SAF 2018a,b)

TU WIEN Change Detection

07/2015 – Present

5.3 GHz

Global

ASCAT

Metop-C

H 119 (H-SAF 2018a,b)

TU WIEN Change Detection

11/2018 – Present

5.3 GHz

Global

2. Upgrade cycle implementation procedure

System upgrade cycles mainly concern the Technical Platform and Operations Framework as well as the C3S Processing Framework. These cycles are timely planned and tested before the final implementation takes place. Implementation upgrades are rolled out to the system during maintenance windows (see Section 4.1) in order to assure smooth operations of the ICDR production without interruptions. Beforehand, system upgrades are tested by making use of the provided test facilities of the corresponding CDR production line. Notifications in the direction of users will be kept minimal as long as no direct implication of the upgrade is given.

2.1. Technical Platform and Operations Framework upgrades

Upgrades related to the Technical Platform and Operations Framework are implemented and tested by the Operations Team. The utilised tools for hardware and software provisioning and configuration (see Section 1.2.2) are fully exploited for upgrade cycles by taking advantage of the source code driven approaches of these tools. With respect to that, hardware setup and software deployments are stored in GitLab under version control. This allows a parallel, automated setup of a fully operational test facility to investigate the foreseen upgrades before deploying them to the operational service. Finally, upgrades will be integrated to the operational service during a maintenance window when all tests in the test facility have been successfully passed.

2.2. C3S Processing Framework upgrades

Processing Framework upgrades are triggered by improved retrieval algorithms and the scientific advancements in the CCI SM processor. All released CCI SM product generated with a specific processor version are verified, validated and relevant algorithm improvements are published in the scientific literature. After a new CCI SM processor version is found suitable, its integration into the C3S production system is started.

2.2.1. Verification and Validation of CCI SM products

Verification and validation of the CCI SM product generated with a specific CCI processor version is performed by four independent teams in the framework of the project. Procedures to verify and validate the products are performed on different spatial scales and in comparison to different soil moisture products available. Results of the CCI SM product verification and validation activities are regularly summarised in the Product Validation and Intercomparison Report (PVIR) published on the CCI SM web portal8.

8 https://esa-soilmoisture-cci.org/ [the URL resource last viewed on 6th September 2023]

2.2.2. Integration of CCI SM product updates into C3S

Product and algorithm updates shall only be integrated into C3S after the verification and validation of the CCI SM products was performed successfully. Any significant algorithm updates shall also be published in scientific journals before they are adapted in the C3S project. Implementations of the algorithm updates will be performed in parallel to the operational service at least two months before a new CDR production cycle is started. Implementation tests provided by the Science Team to assure the correct implementation of the processor have to be passed successfully in order to be considered as ready for operations.

2.2.3. Communication strategy for product updates

After a successful implementation of the upgraded C3S Processing Framework (CPF), the CDR production is performed as scheduled in the production cycle based on the new version. The CDR produced with the updated version of the CPF will be validated in accordance to the Product Quality Assurance Document (PQAD) [D1]. Furthermore, CPF updates require a phase of parallel operations (see Figure 5), for at least three months, of two ICDR versions to be disseminated to the users to enable tests and adoption to the new product version. As of January 2023, two ICDR versions are disseminated in parallel – v201912 and v202012. This enables users to adapt their processing chains in time.

Communication about the release of a new product version will be started another three months before the release through the Climate Data Store (CDS).

Figure 5: Upgrade cycle from one version of the CDR to the next. A phase of parallel operation gives the users enough time to adapt to the new version.

2.2.3.1. Test product availability

During the parallel operation phase and transition to a new CPF, the new CDR product version will be disseminated through the CDS to the users for adoption of their processing chains.

3. Procedures for reprocessing CDR's

The C3S ECV SM consists of a CDR, which represents the archive of the C3S soil moisture product, and the ICDR which is a consistent extension of the CDR. The ICDR products are generated every dekad (approx. 10 days) and extend the CDR of the same version as the ICDR. A new version of the CDR will be produced in the following cases:

  1. Algorithm updates as described in Section 2.
  2. Processing parameter updates.
  3. Addition of new sensors using an existing algorithm.
  4. Change in NRT input products that make a reprocessing necessary.

Generally, the number of versions will be kept to a minimum by pooling the possible changes into yearly releases.

3.1. Processing parameter updates

The processing parameters are regenerated during any CDR reprocessing. An explicit processing parameter update is not necessary if another reason for production of a new version of the CDR exists. In the absence of other changes, the processing parameters will be updated yearly after the release of the most recent CDR version.

3.2. Addition of new sensors

The addition of new sensors triggers the release of a new CDR version.

3.3. Change in input products

A change in input products can happen planned or unplanned. Planned changes include improvements of level 2 soil moisture algorithms either for historical missions or NRT products. Unplanned changes can occur in the ingested NRT products for various reasons and will be handled as described in Section 4.2.5

4. System maintenance and system failures

System maintenance and failures may arise across all system components, but the most likely affect the Technical Platform and/or the Data Management and Streams. Hereafter, procedures applied in case of system maintenance and failures are described.

4.1. System Maintenance

The timeliness of the C3S ICDR production and delivery is 10 days. Maintenance of individual system components will be planned carefully by the Operations Team resulting in an estimate of the needed effort. Based on the delivery and production of the products, the Operations Team will have a maintenance window of 5 days to perform the required actions. Within these 5 days of maintenance the system will adopted or at least will be prepared for maintenance in the next open maintenance window. In general, maintenance of the system to be carried out is not expected to take so long as to impede the ability to deliver the products with a delay of 10 days. If that is the case, we will follow the system failure procedures.

4.2. System Failures

Failures may arise abruptly any time. System failures are likely caused by failures of the Technical Platform of the system rather than other components of the system. Nevertheless, procedures to be applied in case of individual failures will be summarised in the following. Notification to users about system components failures will only happen if they are directly affected, such as product delivery issues. This notification of the users will be done via the CDS which holds the user database for notification. The period of notice will be latest two days before products are expected by the users.

4.2.1. Loss of key staff

At a first response key staff can be replaced by qualified colleagues from within the partner organisation. If this is not possible then the staff will be replaced by external candidates, in the meantime tasks may be distributed among partners to ensure a continuity of project progress.

4.2.2. Non-performance or loss of consortium members

The consortium team have a long standing successful professional relationship, having worked together over various projects for the last 8 years. As such non-performance is not expected. If a consortium member leaves then an adequate replacement will be sought and will be proposed to European Center for Medium-range Weather Forecasts (ECMWF).

4.2.3. Conflict or dispute between team members

All conflicts and disputes will first be addressed and a resolution sought by the project manager. A method for conflict resolution is in place in sub-contracts between EODC and the partners.

4.2.4. Cost overruns

Regular monitoring of the system will allow the project manager and Operations Team to spot and mitigate any costs overruns. If cost overruns cannot be controlled within the project, then ECMWF will be informed and an alternate solution be sought.

4.2.5. Input data stream failure

The product relies upon a suite of earth observation data sets from both active and passive. The loss of a number of input products would only result in a degradation of the quality of the overall product, and not critically stop the production of the CDS product. Only the loss of all input products would result in the inability to generate a product. Nevertheless, users will be informed about the failure via CDS after the first recognition of the issue.

4.2.6. EO Storage failure

Failures related to the EO Storage system are the most critical ones at the moment. Two scenarios of failure may become apparent. First, the disk storage system (tier1) is not accessible, but tier3 the tape library is still available. In that scenario, an additional available disk storage pool used by the Science Integration & Software Development Platform (SIDP) with much less capacity will be used to recall the required data for processing. No action towards the users is needed as long as the product can be delivered in time. In case of a total blackout of the storage system, tier1 and tier3, currently no backup solution is in place. Accordingly, all users will be notified immediately. Missed products will be disseminated on the next possible release cycle of the ICDR.

4.2.7. Processing Facility Failure

Compute resources of the Technical Platform may fail because of damages in the hardware, broken cooling system, etc. In that case, the Operations Team will start to deploy this part of the system on a remote location such as on TU Wien infrastructure to run the required processing there. Users will be informed if a timeliness delivery of the product is critical (2 days before official publication).

4.2.8. Processing Failure

Processing failure may arise after upgrades in the C3S Processing Framework. Such failures are very unlikely, because updates of the framework are well tested beforehand. In case of a failure, the update will be dropped and the previous version of the C3S Processing Framework will be setup again. There is no need to notify the users, because the failure will be resolved immediately and will not affect the user.

User support

A dedicated service desk has been set up, the Copernicus User Support (CUS) team, which provides support to users of the Copernicus Atmosphere Monitoring Service (CAMS) and C3S services at ECMWF. All enquiries about the soil moisture dataset must be submitted through the service desk where it will be dealt with by appropriate persons.

C3S operates a dedicated user support portal9 where customers can submit enquiries using a form (split into “Get our products”, “Licence and invoice”, “Scientific or technical question about our products” and “Problem receiving or accessing our products”). The information from this form is passed to the CUS. Once submitted, the user may add comments or further information to the issue, including responding to questions / requests for additional information from the support team.

In addition to submitting enquires through the portal a knowledge base10 is available to users which can be searched for information.

The Soil Moisture service has a team account with the Copernicus User Support Jira Service Desk System, to provide level 2 user support, i.e. to answer enquiries specific to their products, by direct interaction with the user through the Jira helpdesk. Once a request is sent, the CUS Service team at ECMWF will handle the requests within 8 hours (level 1).

For any scientific and special enquiries that cannot be answered by the CUS team at ECMWF or addressed to the Knowledge Base, the request will be forwarded to the Copernicus User Support Specialists (level-2). Enquiries forwarded to the Copernicus User Support Specialist team will be acknowledged within 3 working days (target 100%) and a notification sent to the user.

In case of specific scientific issues, the enquiries will be channelled to the ECV and data specialist of the C3S2_312a_Lot4 project (under which the Soil Moisture dataset is made available) and should be resolved within 3 working weeks (target 85%).

In each quarter, we aim for User Support satisfaction scoring 3 in 90% of all voluntary based feedbacks by users, with 1 (very unsatisfied) to 5 (very satisfied). We will also list the number of tickets in the Quarterly Report.

9 https://cds.climate.copernicus.eu/cdsapp#!/usersupport  [the URL resource last viewed on 6th September 2023]

10 https://climate.copernicus.eu/help-and-support [the URL resource last viewed on 6th September 2023]

References

Crapolicchio, R., A. Bigazzi, G. De Chiara, X. Neyt, A. Stoffelen, M. Belmonte, W. Wagner, & C. Reimer (2016) The scatterometer instrument competence centre (SCIRoCCo): Project's activities and first achievements, Proceedings European Space Agency Living Planet Symposium 2016, 9-13 May 2016, Prague, Czech Republic, 9-13.

H SAF (2018a). ASCAT Surface Soil Moisture CDR2017 time series 12.5 km sampling – Metop (H113), EUMETSAT SAF on Support to Operational Hydrology and Water Management. http://dx.doi.org/10.15770/EUM_SAF_H_0005 (URL resource last accessed 6th September 2023)

H SAF (2018b). ASCAT Surface Soil Moisture CDR2017-EXT time series 12.5 km sampling – Metop (H114), EUMETSAT SAF on Support to Operational Hydrology and Water Management. https://navigator.eumetsat.int/product/EO:EUM:DAT:0108?query=H114&s=advanced (URL resource last accessed 6th September 2023)

TU Wien (2013). ERS AMI-WS (ESCAT) Surface Soil Moisture Product generated from E1/2-SZ-WNF/UWI-00 dataset. Department of Geodesy and Geoinformation, TU Wien, 2013.


This document has been produced in the context of the Copernicus Climate Change Service (C3S).

The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts, operator of C3S on behalf of the European Union (Delegation Agreement signed on 11/11/2014 and Contribution Agreement signed on 22/07/2021). All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose.

The users thereof use the information at their sole risk and liability. For the avoidance of all doubt , the European Commission and the European Centre for Medium - Range Weather Forecasts have no liability in respect of this document, which is merely representing the author's view.

Related articles