Contributors: B. Calmettes (CLS), G. Calassou (CLS), N. Taburet (CLS)

Issued by: CLS/ B. Calmettes

Date: 06/03/2024

Ref: C3S2_312a_Lot4.WP3-SQAD-LK-v2_202401_LWL_System_Quality_Assurance-v5_i1.0

Official reference number service contract: 2021/C3S2_312a_Lot4_EODC/SC1

Table of Contents

History of modifications

Version

Date

Description of modification

Chapters / Sections

i0.1

15/01/2024

update document to include system information about LWL-S generation

All

i1.0

16/01/2024

Internal review and document finalization, followed by external review, and after minor proofreading edits, publication

All

List of datasets covered by this document

Deliverable ID

Product title

Product type (CDR, ICDR)

C3S Version Number

Delivery date

WP2-FDDP-LWL-CDR-v5

Lake Water Level

CDR

V5.0

31/12/2023

WP2-FDDP-LWL-S-CDR-v1

Lake Water Level - Single-track Lakes

CDR

V1.0

31/12/2023

Related documents

Reference ID

Document

D1

Calmettes, B. et al. (2024). C3S Lake Water Level Version 5.0: Algorithm Theoretical Basis Document. Copernicus Climate Change Service. Document ref. C3S2_312a_Lot4.WP2-FDDP-LK-v2_202312_LWL_ATBD-v5_i1.1.

D2

Calmettes, B. et al. (2023) C3S Lake Water Level Version 5.0: Product Quality Assurance Document. Copernicus Climate Change Service. Document ref. C3S2_312a_Lot4.WP1-PDDP-LK-v2_202306_LWL_PQAD-v5_i1.1.

Acronyms

Acronym

Definition

ATBD

Algorithm Theoretical Basis Document

C3S

Copernicus Climate Change Service

CAMS

Copernicus Atmosphere Monitoring Service

CCI

Climate Change Initiative

CDR

Climate Data Records

CDS

Climate Data Store

CLS

Collecte Localisation Satellites

CNES

Centre National d'Etude Spatiale

CPF

C3S Processing Framework

CSV

Comma Separated Values

CTOH

Center for Topographic studies of the Ocean and Hydrosphere

CUS

Copernicus User Support

DORIS

Doppler Orbitography by Radiopositioning Integrated on Satellite

ECMWF

European Center for Medium-range Weather Forecasts

ECV

Essential Climate Variable

EODC

Earth Observation Data Center

ENVISAT

Environmental Satellite

ERS

European Remote Sensing

ESA

European Space Agency

EUMETSAT

European Organisation for the Exploitation of Meteorological Satellite

FDR4ALT

Fundamental Data Records for Altimetry (ESA)

FTP

Fast Transfer Protocol

GFO

Geosat-Follow-On

GPFS

General Parallel File System

GPGPU

General-purpose Processing on Graphics Processing Units

HAL

Cluster High Performance of CNES

ICDR

Intermediate Climate Data Record

JPL

Jet Propulsion Laboratory

KPI

Key Performance Indicator

L2E-HR

Level-2 Expertise – High-Rate

LEGOS

Laboratoire d'Etudes en Géophysique et Océanographie Spatiales

LK

Lake

LWL

Lake Water Level

LWL-S

Lake Water Level – Single-track Lakes

NASA

National Aeronautics and Space Administration

NOAA

National Oceanic and Atmospheric Administration

NRT

Near Real Time

NTC

Non-Time Critical

OSTM

Ocean Surface Topography Mission

PB

Peta Bytes

PB-F06

Processing Baseline F06

PQAD

Product Quality Assessment Document

RD

Research and Development

SQAD

System Quality Assurance Document

SQL

Structured Query Language

SSALTO

Segment-Sol multi-missions d'ALTimétrie, Orbitographie et localisation précise

SSH

Secure Shell

STC

Short Time Critical

STM

Surface Topography Mission

TCDR

Thematic CDR

TOPEX

TOPography Experiment

US

United States

General definitions

Lake Water Level: Measure of the absolute height of the reflecting water surface beneath the satellite with respect to a vertical datum (geoid) and expressed in metres.

Altimeter: A device that measures altitude, the distance of a point above sea level. Radar altimetry measures the two-way travel time of a radar pulse between the satellite antenna and the Earth's surface at the nadir of the spacecraft, from which an estimate of Lake Water Level can be derived and delivered to users as a dataset product.

Single-track lake:  is a lake observed over several cycles by a single track of an altimetry mission satellite. Those lakes are usually small, with a mean surface area of 13.44 km2 in the Lake Water Level – Single-track Lakes (LWL-S) dataset v1.0

Scope of the document

This document is the System Quality Assurance Document (SQAD) for the Copernicus Climate Change Service (C3S) Lake Water Level (LWL) Service. It describes the systems used to provide LWL and Lake Water Level – Single-track Lakes (LWL-S) products to the C3S Climate Data Store (CDS), and all elements (e.g. teams) that contribute to the processing chain, including all interfaces to C3S, and to Research and Development (RD) component, the hardware used in the processing chain, the procedures to implement new data cycles and to reprocess the products, and those applied in case of system failures or anticipated maintenance work, as well as information on user support.

Executive summary

The C3S Lake production system (C3S ECV LK – LWL) provides an operational Lake Water Level (LWL)  and Lake Water Level - Single-track (LWL-S) service, generating lake water level climate datasets for a wide variety of users within the climate change community. A system of people, hardware, software, and interfaces ensure reliable delivery of high quality data to C3S users, which is described in this document. The Copernicus Climate Change Service (C3S) Lake Water Level production system (C3S ECV LK – LWL) is responsible for the operational provisioning of the LWL parameter within the Lake Essential Climate Variable (ECV) for the C3S. The mission of the system is the timely production of a consistent, long-term, multi-mission and multi-sensor global lake water level data set complying with the defined key performance indicators (KPIs). The system is semi-automated, intelligible and cost-effective to foster the interoperability of the individual system elements and teams.

In terms of people, a dedicated team supports and maintains the system. It is divided into a science team, a system development team and an Operations team. The science team directs the scientific development of the service and provides dedicated scientific support to the users. The system development team is responsible for the implementation of the upgrades and product/service developments, the maintenance and the validation of the Technical Platform. The Operations team is responsible for the Operational Framework. The responsibilities of the Operations team, with respect to system maintenance and system failures, are summarised in a list of actions to trigger in case that they are required. Together, these teams operate  and run the hardware, software, algorithm development and quality control that underpins data generation, and delivery.

This System Quality Assurance Document (SQAD) details the integral components of the C3S ECV LK – LWL production system and their functions and interfaces (Section 1). The implementation procedure of the upgrade cycle is given in Section 2 while Section 3 describes the procedures for reprocessing of Climate Data Records (CDRs). Section 4 details procedures that are applied in case of system maintenance or failures and Section 5 provides information on the User support.

Please note, that unless stated otherwise, the LWL stands for both LWL and LWL-S products in the following sections.

1. System overview

1.1. System elements and interfaces

The system for generating data, delivering it to the C3S, and providing help and support to users is comprised of a number of different elements. These elements are people (e.g. the science team), hardware and software (e.g. the technical platform). They, and their interfaces with C3S are described in the following sections.

1.1.1. Components of the system

Integral components of the C3S ECV LK – LWL are:

  • the Organizational Framework consisting of
    • the LWL Science team
    • the LWL system development team
    • the LWL Operations team
  • the Technical Platform and the Operations Framework
  • the C3S Processing Framework (CPF)
  • Data Management and Streams

These components, their functions and interfaces within the C3S ECV LK – LWL production system will be discussed in the following sections.

1.1.2. Organizational Framework

Three teams are cooperating within the organizational framework in order to fulfil the mission objective of the system.

  • The Science Team, composed by scientist from Laboratoire d'Etudes en Géophysique et Océanographie Spatiales (LEGOS) and Collecte Localisation Satellites (CLS) (France), oversees the physical reliability and validity of the LWL generated by the systems. Consequently, the function of the Science Team is to design scientific algorithms capable of producing a long-term consistent, multi-mission global LWL product, in accordance with the defined data key performance indicators.
  • The System Development Team develops and maintains the scientific algorithms based on the design of the science team and integrates them in the processing baselines. It validates the implementation in accordance with the defined data key performance indicators and produces a new version of the systems that they deliver to the corresponding Operations Team (CLS1, France).
  • The Operations Team, based in CLS (France) manages the system and its individual components and takes control of the regular production respectively of the LWL variable. With respect to that, the Operations Team performs the orchestration, provisioning and configuration of the technical platform as well as the setup and maintenance of the Operations Framework. Members of the Operations Teams exclusively have access to the Technical Platform and Operations Framework. Inputs provided by the System Development Teams, such as new versions of the processing baselines, are picked up by the Operations Teams and are revised in an iterative and cooperative way before those are integrated in the operational C3S Processing Framework.

All teams may be involved in providing substantive support to users in response to questions posed by the C3S user support service.

1 https://www.cls.fr/en/ [the URL resource last viewed on 6th March 2024]

1.2. Hardware, supercomputers and computing

1.2.1. Technical Platform and Operations Framework

The LWL Technical Platform and Operations Framework is composed of a set of hardware and software components interacting with each other.

1.2.1.1. Hardware component

The C3S LWL Technical Platform is realised at CLS and Centre National d'Etude Spatiale (CNES)2 providing two major hardware components. These hardware components are: the CLS cluster: nine batch servers / 216 cores (432 threads) 128GB memory / server; the CNES cluster (HAL): 300Tflops, 380 batch servers / 8400cores, 128GB memory / server, 6,2 PB GPFS / 200TB burst buffer/ 100GBs bandwidth Infiniband, low latency network, 4 GPGPU Nvidia Volta V100The HAL cluster is used to gather and enhance the historical altimetry data thanks to its significant storage capability and computing power. The historical altimetry databases (L2E-HR) on the server are maintained and operated with a CLS software and database proprietary format directly compatible with LWL system. The CLS cluster is used to copy and store only the relevant input data from the altimetry L2E-HR databases on the HAL cluster. This data is stored temporarily on a dedicated partition (netapp4-L2P-HYDRO, 3To) for the Operations Team's use. The partition also stores the relevant missing ancillary data and houses the operational C3S LWL Processing Framework.

2 https://cnes.fr/en [the URL resource last viewed on 6th March 2024]


1.2.1.2. Software component

The Operations Framework is based on a source code library base on GitHub3 and its web-based repository manager GitLab4, holding all code and configuration fragments needed to run the operational service. Writing access to the GitLab code repository is restricted to the members of the LWL System Development Team. Reading access to the GitLab code repository is extended to the members of the C3S LWL Operations Team. Specific versions of code packages can be cloned or downloaded to the Technical Platform. This process is automated by making use of a CLS overlayer software to the Git tools. The documentation platform of all packages is hosted on a dedicated Microsoft SharePoint repository with restricted access to the Science Team, the System Development Team and the Operations Team.

System operations are done by members of the Operations Team. Team members have unrestricted access to the Technical Platform and the Operations Framework as a single Technical User. This Technical User, with username lake_exp, handles the entire operational service of the C3S ECV LK - LWL. Secure access to the Technical Platform and Operations Framework is exclusively given via Secure Shell (SSH) using public-key cryptography for authentication.

3 https://github.com/ [the URL resource last viewed on 6th March 2024]

4 https://about.gitlab.com [the URL resource last viewed on 6th March 2024]

1.3. C3S Processing Framework

The C3S Processing Framework (CPF) for LWL is based upon the scientific base of the THEIA/Hydroweb project5 and is named Hysope. This means that the CPF uses the Hysope processor of a specific version for producing both the C3S LWL and C3S LWL-S products with data accessed from different repositories: at the CTOH (Center for Topographic studies of the Ocean and Hydrosphere)6, at HAL in CNES and at the CLS cluster. Pre- and post-processing needed to get the input data streams into the correct format to be handled by the Hysope processor or to re-format the data to the C3S specifications is added around the Hysope processor core. Figure 1 provides a high-level look at the components of the framework, also described in Table 1.

5 https://www.theia-land.fr/en/hydroweb/ [the URL resource last viewed on 6th March 2024]

6 https://www.legos.omp.eu/ctoh/ [the URL resource last viewed on 6th March 2024]

 


Figure 1: Overview of the elements and interfaces that comprise the C3S Processing Framework used to create C3S LWL data. The boxes in yellow indicate the data providers. The grey cylinders represent databases accessed by the data providers or by one of the processors (represented in green). The science team provides parameter files via a ftp server (in orange) to Hysope processor, which updates the CLS server before the last processor step that generates the output file in C3S format. The direction of the arrows indicates the flow of data transfers.

Table 1: LWL C3S Processing Framework Components. 

Component Name

Component Description

FTP

Fast Transfer Protocol Exchange Platform with Science Team

Database

SQL database for interactive use by the Hysope processor in NRT mode

CLS server

Local Disk Space with output operational products in CSV (.csv) format

1.3.1. Interface to Hysope processor

The original Hysope processor was designed to ingest Level-2 altimetry data. The input format can either be the NetCDF (.nc) official format delivered by the European Space Agency (e.g., cophub7) or the CLS proprietary binary format. This latter functionality allows the Hysope processor to ingest the historical altimetry data stored in the database L2E-HR, in the CLS format, housed by and maintained on the HAL cluster.

Figure 2 shows how the input data stream interacts with the Hysope processor.

Figure 2: C3S LWL input data streams (present and future). NTC for Non Time Critical data and STC for Short Time Critical data.

The Hysope processor was designed to run in different modes. In the “reprocessing” mode, the Hysope processor computes processing parameters and produces the most consistent lake water level CDR based on all available input datasets (historic and current missions). The “Near-Real-Time” (NRT) mode of the Hysope processor uses these processing parameters to generate the Intermediate Climate Data Record (ICDR), taking into account the latest available input data from all current missions, with less computational effort compared to the “reprocessing” mode. Processing parameters are generated during each reprocessing cycle. The parameters are the geographical extraction zones for each lake, empirical correction of the high frequency variations of the geoid and inter-mission biases. These parameters are stored for each sensor and each lake and used both in “reprocessing” and “NRT” modes of the processor. Figure 1 shows how the parameter files interact with the Science Team and the Hysope processor via the Fast Transfer Protocol (FTP) component and are archived on the CLS server component. More details about the algorithm and the used parameters can be found in the Algorithm Theoretical Basis Document (ATBD) [D1].

7 https://cophub.copernicus.eu/ [the URL resource last viewed on 6th March 2024]


1.3.2. Post Processing

The Hysope processor can produce for each lake either a Comma Separated Values (CSV) or GeoJSON file in both modes. Consequently, the C3S post processing block converts the format into the required NetCDF4 format and provides the data to the Earth Observation Data Center (EODC) infrastructure that will then provide the data to the Climate Data Store (CDS).

1.4. Data Management and Streams

Data used in the C3S ECV LK LWL can be divided into historical and NRT inputs. Historical data are level-2 altimetry products from already decommissioned satellite missions generated in one processing cycle. These datasets are reprocessed on a regular basis by space agencies to enhance them with the state-of-the-art algorithms. The data archive is available on the disk storage of CLS cluster and on the CTOH (Center for Topographic studies of the Ocean and Hydrosphere), data center of the Science Team. The data is archived and is complemented by regular backups of the data.

For each processing cycle of the CDR and Test CDR, the generated data will be placed in separate directories to assure data integrity of the different versions. Furthermore, data produced by test facilities of the system will be completely separated from the operational data stream for the same reason. The latest version of an LWL CDR and ICDR will reside on disk storage (netapp4) as long as the data stream is operational. Outdated, older versions of the LWL product with its corresponding processing parameters are archived on the same disk storage (netapp4) in a different repository.

Movement and deletion of data can only be done by the Operations Team and is closely coordinated with the Science Team.

In terms of interfaces, NRT data streams required for the operational production of the ICDR are regularly monitored by the Operations Team by utilising workflow routines implemented in Nagios8. These checks are executed beforehand of each ICDR production run, raising an error if a data stream is broken.

8 https://www.nagios.org/ [the URL resource last viewed on 6th March 2024]

1.4.1. Active input streams

Active input data streams are based on observations from a series of altimeters on-board historical and current international satellite missions (Figure 3). Algorithms to derive LWL from altimetry have been developed and are further research by Laboratoire d’Etudes en Géophysique et Océanographie Spatiales (LEGOS) (Crétaux et al 2006, Crétaux et al 2011, Crétaux et al 2016).

Figure 3. Current and historical altimetry missions which comprise the altimetry constellation which could be of use to LWL data production. For LWL-S only data from current missions (Sentinel -3A/B and Sentinel-6A) and Jason-3 are used.  (source: www.aviso.altimetry.fr)9.

9 For more information see https://doi.org/10.24400/527896/A02-2022.001 [URL resource last viewed on 6th March 2024]

1.4.1.1. Sentinel-6A

Sentinel-6A, also known as Sentinel-6 Michael Freilich, is an ongoing altimetry radar mission, successor of TOPography Experiment (TOPEX)/Poseidon, Jason-1, Jason-2 and Jason-3 (see Section 1.4.2.1). The Sentinel-6A NRT Level-2 data is operationally collected on the CLS cluster through the CNES multi-missions ground segment SSALTO data server (Segment-Sol multi-missions d'ALTimétrie, Orbitographie et localisation précise) housed by CNES.

1.4.1.2. Sentinel-3A/B

The Sentinel-3 satellites fit into the Copernicus programme. Sentinel-3A was launched in 2016. It has been joined by Sentinel-3B in 2018 for a 4-month tandem phase before it drifted to its nominal orbit, an interleaved orbit with respect to Sentinel-3A.

The Sentinel-3 Short Time Critical (STC) Level-2 Land products are operated by European Space Agency (ESA) and broadcast on the Copernicus hub. The data is operationally downloaded by the Operations Team and stored on the CLS cluster back up disk (netapp3).

1.4.2. Passive input streams

In the individual sections for the sensors, an overview is given on what specific datasets are currently available for ingestion into the C3S processing system and in case of NRT processing, how this is set up.

1.4.2.1. TOPEX/Poseidon and Jason

The TOPEX/Poseidon satellite was launched in 1992 with the objective of "observing and understanding the ocean circulation". Established as a joint project between the National Aeronautics and Space Administration (NASA), the US space agency, and CNES, the French space agency, it carries two radar altimeters and precise orbit determination systems. The original design was intended to monitor oceans and it also measured inland water levels. The TOPEX/Poseidon data is stored on the CTOH cluster for future use in the Hysope processor. The reprocessing of TOPEX/Poseidon was finished in 2022 and latest deliverables are intended to be delivered in the first quarter of 2023.

Developed jointly by CNES and NASA, Jason-1 was launched in 2001 and is the follow-on of TOPEX/Poseidon. Its unchanged orbit, compared to that of TOPEX/Poseidon, allows a continuous acquisition of measurements and, thus, improved stability of the C3S LWL product. The Jason-1 data is stored on the CTOH cluster. The 2016 reprocessing is stored on the HAL cluster for future use in the Hysope processor.

Jason-2/OSTM (Ocean Surface Topography Mission) takes over and continues TOPEX/Poseidon and Jason-1 missions in 2008, in the frame of a cooperation between CNES, EUMETSAT (European Organisation for the Exploitation of Meteorological Satellite), NASA and NOAA (National Oceanic and Atmospheric Administration). It carries the same kind of payload as its two predecessors, for a high-accuracy altimetry mission: a Poseidon-class altimeter, a radiometer and three location systems. The orbit is also the same. The Jason-2 data is stored on the HAL Cluster for future use in the Hysope processor.

Jason-3 followed on Jason-2 seamlessly on the same orbit with the same payload. It was launched in 2016 carrying a typical suite of altimetry mission instruments to extend data records compiled by TOPEX/Poseidon, Jason-1 and Jason-2. Since April 2022, Jason-3 is on an interleaved orbit.

The launch of Sentinel-6A (Sentinel-6 Michael Freilich) and its orbit in tandem with Jason-3 for 12 months, ensures the continuity of the Jason-3 time-series of data. This satellite is part of the Copernicus programme and is a result of international cooperation between ESA, EUMETSAT, European Union, NOAA, CNES and NASA/JPL (Jet Propulsion Laboratory).

1.4.2.2. Envisat

Envisat (Environmental Satellite) is the follow-on to European Remote Sensing ERS-1 and ERS-2 (not used yet in C3S LWL product). Devoted to environmental studies, and climate change in particular, its mission was to observe Earth’s atmosphere and surface. Built by ESA, Envisat carried 10 complementary instruments for observing parameters ranging from the marine geoid to high-resolution gaseous emissions. Among these instruments are a radar altimeter, and the Doppler Orbitography by Radiopositioning Integrated on Satellite (DORIS), an orbitography and precise location system. A reprocessing of Envisat altimetry Level-2 product has been released in 2016, upgrading standards to the state-of-the-art. This reprocessed data has been acquired and checked by CLS before storage in internal databases, and will be considered for a reprocessing of the LWL product to improve precision, accuracy and stability. Additionally, in the framework of the FDR4ALT project10, founded by ESA, a new reprocessing of Envisat altimetry Level-2 products is ongoing. This reprocessing includes, for the first time, inland water thematic data products and is expected to improve the quality of LWL products during the period 2002-2009. Data from Envisat has been generated and is already available.

10 Fundamental Data Records for Altimetry: https://www.fdr4alt.org/ [the URL resource last viewed on 6th March 2024]

1.4.2.3. Geosat-Follow-On

GFO, Geosat Follow-On, was launched in February 1998. Its main mission is to provide real-time ocean topography data to the US Navy thanks to its radar altimeter. It also provided inland water measurements. The data can be accessed through NOAA and has been collected by the CTOH data center.

2. Upgrade cycle implementation procedure

System upgrade cycles mainly concern the Technical Platform and Operations Framework as well as the C3S Processing Framework. These cycles are timely planned and tested before the final implementation takes place. Implementation upgrades are rolled out to the system during maintenance windows, in order to assure smooth operations without interruptions. Beforehand, system upgrades are tested by making use of the provided test facilities of the corresponding CDR production line. Notifications in the direction of users will be kept minimal as long as no direct implication of the upgrade is given.

2.1. Technical Platform and Operations Framework upgrades

Upgrades related to the Technical Platform and Operations Framework are implemented and tested by the Operations Team. The utilised tools, mainly GitLab described in Section 1.2.1.2, for software provisioning and configuration are fully exploited for upgrade cycles by taking advantage of the source code driven approaches of these tools. With respect to that, software deployments are stored in GitLab under version control. This allows a parallel, automated setup of a fully operational test facility to investigate the foreseen upgrades before deploying them to the operational service. Finally, upgrades will be integrated in the operational service during a maintenance window when all tests in the test facility have been successfully passed.

2.2. C3S Processing Framework upgrades

Processing Framework upgrades will be triggered by improved retrieval algorithms and the scientific advancements in the Climate Change Initiative (CCI) LK LWL processor. All released CCI LK LWL products generated with a specific processor version are verified, validated and relevant algorithm improvements are published in the scientific literature. After new CCI lake processor versions are found suitable, their integration into the C3S production system is started.

2.2.1. Verification and Validation of CCI LK LWL products

Verification and validation of the CCI LK LWL products generated with specific CCI processors versions will be performed in the framework of the CCI LK project. Procedures to verify and validate the products will be performed on different spatial scales and, in comparison to different LWL products available. Results of this verification and validation activities will be regularly summarised in the Product Validation Plan published on the CCI LK webportal11.

11 https://climate.esa.int/media/documents/CCI-LAKES-0030-PVP_v1.3.pdf  [the URL resource last viewed on 6th March 2024]

2.2.2. Integration of CCI LK LWL products updates into C3S

Products and algorithm updates shall only be integrated into C3S after the verification and validation of the CCI LK LWL products is performed successfully. Implementations of the algorithm updates will be performed in parallel to the operational service at least two months before a new CDR production cycle is started. Implementation tests provided by the Science Team to assure the correct implementation of the processor have to be passed successfully in order to be considered as ready for operations.

2.2.3. Communication strategy for product updates

After a successful implementation of the upgraded C3S Processing Framework (CPF), the CDR production is performed as scheduled in the production cycle based on the new version. Once per year, a Test CDR will be produced with the updated version of the CPF. Six months later, a CDR will be generated based on the Test CDR with a sixth-month update of the product. It will then be validated in accordance with the Product Quality Assurance Document (PQAD) [D2]. The previous product version is stopped and are preserved at CLS and at C3S repository.  Therefore, the new version takes over the operational data stream. All products versions and description are included in the ECV inventory12. Communication about the release of a new product version will be started another three months before the release through the CDS.

12 https://climatemonitoring.info/ecvinventory/ [the URL resource last viewed on 6th March 2024]

3. Procedures for reprocessing CDR's

A CDR consists of a Thematic CDR (TCDR), which represents the archive of the C3S LWL product, and the ICDR which is a consistent extension of the TCDR. The ICDR products are generated every six months and extend the TCDR of the same version as the CDR. A new version of the CDR will be produced in the following cases:

  1. Algorithm updates as described in Section 2
  2. Processing parameter updates
  3. Addition of new sensors using an existing algorithm
  4. Change in NRT input products that make a reprocessing necessary

The number of versions will be kept to a minimum by pooling the possible changes into yearly releases.

3.1. Processing parameter updates

The processing parameters are regenerated during any CDR reprocessing. An explicit processing parameter update is not necessary if another reason for production of a new version of the CDR exists. In the absence of other changes, the processing parameters will be updated yearly after the release of the most recent CDR version.

3.2. Addition of new sensors

The addition of new sensors triggers the release of a new CDR version.

3.3. Change in input products

A change in input products can happen, planned or unplanned. Planned changes include improvements of level 2 water level algorithms either for historical missions or NRT products. Unplanned input data stream failures can occur in the ingested NRT products for various reasons and will be handled as described in Section 4.2.5.

4. System maintenance and system failures

System maintenance and failures may arise across all system components, but most likely they affect the Technical Platform and/or the Data Management and Streams. Hereafter, procedures applied in case of system maintenance and failures are described.

4.1. System Maintenance

The timeliness of the C3S LK LWL/LWL-S ICDR production and delivery is six months. Maintenance of individual system components will be planned carefully by the Operations Team resulting in an estimate of the needed effort. Based on the delivery and production of the products, the Operations Team will have a maintenance window of three months to perform the required actions. Within these three months of maintenance, the required changes in the system will be adopted or at least the system will be prepared for maintenance in the next open maintenance window.

4.2. System Failures

Failures may arise abruptly at any time, but considering the 6-month timeliness of the products, failures can be treated without any major delay in the production. System failures are likely caused by failures of the Technical Platform of the system rather than other components of the system. Nevertheless, procedures to be applied in case of individual failures will be summarised in the following sub-sections. Notification to users about system components failures will only happen if users are directly affected, such as with product delivery issues. This notification of the users will be done via the CDS which holds the user database for notification. The period of notice will be 10 days at the latest before products are expected by the users.

4.2.1. Loss of key staff

As a first response to loss of key staff, staff can be replaced by qualified colleagues from CLS or LEGOS. If this is not possible then the staff will be replaced by external candidates. In the meantime tasks may be distributed among partners to ensure a continuity of project progress.

4.2.2. Non-performance or loss of consortium members

The consortium team has a long-standing successful professional relationship, having worked together over various projects for the last 10 years thus such non-performance is not expected. If a consortium member leaves, then an adequate replacement will be sought and will be proposed to ECMWF (European Center for Medium-range Weather Forecasts).

4.2.3. Conflict or dispute between team members

All conflicts and disputes will first be addressed, and a resolution sought by the service manager. A method for conflict resolution is in place in subcontracts between EODC and the partners.

4.2.4. Cost overruns

Regular monitoring of the LWL system will allow the project manager and Operations Team to spot and mitigate any costs overruns. If cost overruns cannot be controlled within the project, then ECMWF will be informed, and an alternate solution will be sought.

4.2.5. Input data stream failure

The product relies upon a suite of earth observation datasets. The loss of a number of input products would only result in a degradation of the quality of the overall LK products, and not critically stop the production of the CDS product. Only the loss of all input products would result in the inability to generate a product. Nevertheless, users will be informed about the failure via CDS after the first recognition of the issue to adopt for and be prepared.

4.2.6. Cluster failure

Failures related to the CLS or HAL clusters are the most critical ones at the moment. (1) If the CLS cluster blacks out but not the HAL cluster, the historical altimetry data will be still available on the HAL cluster and on CLS backup system on band tapes. CLS has a back-up cluster based on the network of staff computers that may be used to recover altimetry data from the HAL cluster and from the system backups on band tapes. (2) If the HAL cluster blacks out, the HAL cluster data is also backed up on band tapes at CNES and can be recovered. No action towards the users is needed as long as the product can be delivered on time. In case of a total blackout of the storage system for CLS and HAL clusters and the corresponding band tapes, currently no backup solution is in place. Accordingly, all users will be notified immediately. Missed products will be disseminated on the next possible release cycle of the ICDR.

4.2.7. Processing Facility Failure

Computing resources of the Technical Platform may fail because of damage to the hardware (e.g., broken cooling system, etc). In this situation, the LWL Operations Team will start to deploy this part of the system on the backup CLS cluster solution (or the CNES HAL cluster solution if necessary) to run the required processing there. Users will be informed if a timely delivery of the product is critically affected (10 days before official publication).

4.2.8. Processing Failure

Processing failure may arise after upgrades in the C3S Processing Framework. Such failures are very unlikely, because updates of the framework are well tested beforehand. In case of a failure, the update will be dropped, and the previous version of the C3S Processing Framework will be re-setup. There will be no need to notify the users in such situations, because the failure will be resolved immediately and will not affect the user.

5. User support

A dedicated service desk has been set up, the Copernicus User Support (CUS) team, which provides support to users of the Copernicus Atmosphere Monitoring Service (CAMS) and C3S services at ECMWF. All enquiries about the LWL and LWL-S datasets must be submitted through the service desk where it will be dealt with by appropriate persons.

C3S operates a dedicated user support portal13 where customers can submit enquiries using a form (subdivided into “Get our products”, “Licence and invoice”, “Scientific or technical question about our products” and “Problem receiving or accessing our products”). The information from this form is passed to the CUS. Once submitted, the user may add comments or further information to the issue, including responding to questions/requests for additional information from the support team.

In addition to submitting enquires through the portal, a knowledge base is available to users which can be searched for information.

Once a request is sent, the Copernicus User Support (CUS) Service team at ECMWF will handle the requests within 8 hours (level 1).

For any scientific and special enquiries that cannot be answered by the CUS team at ECMWF or addressed to the Knowledge Base, the request will be forwarded to the Copernicus User Support Specialists (level-2).

Enquiries forwarded to the Copernicus User Support Specialist team will be acknowledged within 3 working days (target 100%) and a notification sent to the user. In case of specific scientific issues, the enquiries will be channelled to the ECV and data specialist of the C3S2_312a_Lot4 project and should be resolved within 3 working weeks (target 85%). In each quarter, we aim for a User Support satisfaction rating of 3 or above in 90% of all voluntary based feedbacks by users, with 1 (very unsatisfied) to 5 (very satisfied). We will also list the number of tickets in the Quarterly Report to the C3S team in the ECMWF.

13 https://cds.climate.copernicus.eu/cdsapp#!/usersupport  [the URL resource last viewed on 6th March 2024]

References

Crétaux, J. F., & Birkett, C. (2006). Lake studies from satellite radar altimetry. Comptes Rendus Geoscience, 338(14-15), 1098-1112.

Crétaux, J. F., Jelinski, W., Calmant, S., Kouraev, A., Vuglinski, V., Bergé-Nguyen, M., ... & Maisongrande, P. (2011). SOLS: A lake database to monitor in the Near Real Time water level and storage variations from remote sensing data. Advances in space research, 47(9), 1497-1507.

Crétaux, J. F., Abarca-del-Río, R., Berge-Nguyen, M., Arsen, A., Drolon, V., Clos, G., & Maisongrande, P. (2016). Lake volume monitoring from space. Surveys in Geophysics, 37(2), 269-305.


This document has been produced in the context of the Copernicus Climate Change Service (C3S).

The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts, operator of C3S on behalf of the European Union (Delegation Agreement signed on 11/11/2014 and Contribution Agreement signed on 22/07/2021). All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose.

The users thereof use the information at their sole risk and liability. For the avoidance of all doubt , the European Commission and the European Centre for Medium - Range Weather Forecasts have no liability in respect of this document, which is merely representing the author's view.

Related articles