Contributors: Christoph Reimer (CRe, EODC GmbH), Richard Kidd (RK, EODC GmbH), Robin van der Schalie (RvdS, Vandersat), Christoph Paulik (CPa, Vandersat)
Issued by: EODC GmbH/Richard Kidd
Date: 27/04/2021
Ref: C3S_312b_Lot4.D1.SM.1-v3.0_System_Quality_Assurance_Document_i1.0
Official reference number service contract: 2018/C3S_312b_Lot4_EODC/SC2
History of modifications
List of datasets covered by this document
Related documents
Acronyms
Scope of the document
The System Quality Assurance Document outlines the architecture of the C3S Soil Moisture Production System with a detailed description of the involved system elements and their dependencies. The document aims to provide recipes to mitigate and handle possible scenarios affecting the operational service provided by the system.
Executive summary
The C3S Soil Moisture (SM) production system (C3S ECV SM PS) provides an operational soil moisture service, generating soil moisture climate data sets for a wide variety of users within the climate change community. Two teams support and maintain the system. The science team directs and implements the scientific evolution of the service and provides dedicated scientific support to the users. The Operations Team is responsible for the Technical Platform and the Operational Framework. The responsibilities of the Operations Team, with respect to system maintenance and system failures, are summarized in a list of actions to be triggered in case they are required.
1. System overview
The C3S Soil Moisture (SM) production system (C3S ECV SM PS) is responsible for the operational provisioning of the soil moisture Essential Climate Variable (ECV) for the Copernicus Climate Change Service. The mission of the system is the timely production of a consistent, long-term, multi-mission, global soil moisture data set complying with the defined key performance indicators (KPI’s) [D3]. The intention of the system concept was to create a semi‑automated, intelligible, reproducible and cost‑effective system to foster the interoperability of the individual system elements and teams. Integral components of the C3S ECV SM PS are:
- the Organizational Framework consisting of
- the Science and
- the Operations team
- the Technical Platform and the Operations Framework
- the C3S Processing Framework (CPF), and
- Data Management and Streams
These components, their functions and interfaces within the C3S ECV SM production system will be discussed in the following.
1.1. Organizational Framework
Two teams have been elected within the Organizational Framework in order to fulfil the mission objective of the system. At first, the Science team is in charge of the physical reliability and validity of the soil moisture ECV product generated by the system. As a consequence, the function of the Science Team is to develop and maintain scientific algorithms capable to produce a long-term consistent, multi‑mission global soil moisture product in accordance with the defined data key performance indicators [D3]. Algorithms provided by the Science Team are source code packages to be integrated in the C3S Processing Framework (CPF). These source code packages are exchanged with the C3S ECV SM PS by utilising a well-defined interface realised by shared code repository (see Section 1.2). In addition, the Science Team is responsible to perform regular validation activities of the various SM ECV products released to the Copernicus Climate Change Service through the system.
The Operations Team, on the other hand, is in charge of managing the system and its individual components and takes control of the regular production of the soil moisture ECV. With respect to that, the Operations Team performs the orchestration, provisioning and configuration of the technical platform as well as the setup and maintenance of the Operations Framework (see Section 1.2). Members of the Operations Team exclusively have access to the Technical Platform and Operations Framework. Access by individuals outside of the Operations Team to these components of the system is not foreseen, in order to reduce and mitigate unpredictable system failures. Inputs provided by the Science Team are picked up by the Operations Team and are revised in an iterative and cooperative way before those are integrated in the operational C3S processing framework.
1.2. Technical Platform and Operations Framework
The Technical Platform and Operations Framework is composed of a set of hardware and software components interacting with each other. In the following a detailed description of the individual components will be given.
1.2.1. Technical Platform
The Technical Platform is realised at the EODC providing three major hardware components fully embedded in the C3S ECV SM CPS. These hardware components are:
- the EODC Cloud platform,
- EO Storage, and
- the Vienna Scientific Cluster 3 (VSC3)
EO Storage is the centrepiece of the Technical Platform, accessible from both compute environments, EODC Cloud and VSC3, dedicated to run the different production cycles of the soil moisture ECV. EO Storage is a multi-tier storage system composed of three tiers. The first tier of the storage system is a disk storage cluster (tier1) specifically design for high performance computing (HPC). The second storage tier is a slow on-demand tape storage pool (tier2) accompanied by the third tier a tape library (tier3). At the current stage of the C3S ECV SM CPS only tier1 and tier3 are considered (see Section 1.4). The disk storage (tier1) is foreseen to hold the entire archive of data required for the production of the latest version of the soil moisture ECV (CDR and ICDR). Outdated, older versions of the soil moisture ECV (CDR and ICDR) with their corresponding input and auxiliary data are preserved in the tape library.
During the conceptual phase of the system it was recognised that two distinct processing demands exist, related to the production of the soil moisture CDR and ICDR. It was decided to run the different production cycles, of the CDR and ICDR, in diverse compute environments fulfilling the processing demands and optimise the processing costs (Figure 1). As a result, the production of the soil moisture CDR, with the demand of temporal, largescale compute resources, is carried out on the VSC-3. The benefit of using the VSC-3 is the scalability of the processing job, being able to parallelise individual tasks on thousands of CPUs. VSC-3 is a slurm1 based compute cluster for batch processing jobs. Jobs are submitted to a job queue and executed in sequence as soon as compute resources are available. Non-time critical processing jobs, such as the production of the CDR, are a perfect application for such large-scale compute clusters by minimising compute costs.
Complementary, the ICDR generation, with the demand of highly available, smallscale compute resources, is performed on the EODC Cloud platform. This platform is devised as a virtualised, highavailability compute cluster operated as a private cloud infrastructure based on OpenStack2. The C3S ECV SM PS is incorporated in OpenStack as a particular tenant to guarantee a certain level of security and encapsulate the system from other cloud users and applications. Within the OpenStack tenant a small-scale compute cluster is setup for the operational production of the ICDR. The current setup of the cluster comprises 1 master (4 vCPUs, 8 GB RAM), 4 worker nodes (2 vCPUs, 4 GB RAM) and 1 Level 2 SM processor node (4 vCPUs, 8 GB RAM). The EODC Cloud allows to elastically scale compute resources on demand so that each of these nodes can be replicated to function as a test facility. This cluster is complemented by additional compute instances providing basic services to the C3S ECV SM PS such as a Continuous Integration server and a software package delivery server.
Figure 1: Technical Platform of the C3S ECV SM PS
1.2.2. Operations Framework
The Operations Framework is realised as services and workflows to manage the timely production and delivery of the soil moisture ECV, integrating and building upon the Technical Platform. The major objective of the Operations Framework concept was the creation of an automated, intelligible and reproducible system component. Therefore, innovative IT management tools have been introduced to automate and document the orchestration, provisioning and configuration of the framework, especially with focus on the time critical creation of the ICDR. One of these tools is Terraform3, used to orchestrate the compute cluster on the EODC Cloud platform. Terraform allows to build, change and versioning infrastructure safely and efficiently across multiple data centres. One of the key features of Terraform is to describe Infrastructure as Code (IaC) in a high-level configuration syntax. As a result, a blueprint of the ICDR operation facility is created with this tool to share and re-use in other datacentres and/or with cloud providers. The benefits of this tool for operations are:
- Fast recovery of Technical Platform after hardware system failures.
- Transferability of Technical Platform to other datacentres in case of black-out.
- Changes in the Technical Platform are documented and versioned like source code and therefore reproducible.
Furthermore, the configuration management tool Ansible4 is utilized in the Operations Framework to automate deployments of software packages and tools used in the C3S ECV SM PS. Ansible enables a consistent configuration and provisioning of the ICDR test and operations facilities (compute clusters) in a reproducible way using a human-readable data serialization language (YAML). Each associated node in the cluster is consistently configured with the same technical users and groups, system libraries and required software packages to perform the ICDR generation. Besides a consistent configuration of individual resources, Ansible provides the advantage:
- Versioning control of changes in the configuration individual compute nodes
- Documentation of software configurations in an intelligible way (YAML syntax)
- Reproducible setup of Operations Framework
- Mitigation of software configurations errors
The centrepiece of the Operations Framework is a source code library base on Git5 and its webbased repository manager GitLab6 , holding all code and configurations fragments needed to run the operational service. Access to the GitLab code repository is restricted to members of the C3S soil moisture project consortium. Specific version of code packages can be cloned or downloaded to the Technical Platform. This process is automated by making use of Ansible and the software package delivery service "pypi". Furthermore, GitLab serves as the documentation platform of all packages by making use of the integrated WIKI platform.
System operations is done by members of the Operations Team. Team members have unrestricted access to the Technical Platform and to the Operations Framework as a single Technical User. This Technical User, with username c3s of the group c3s, handles the entire operational service of the C3S ECV SM PS. In order to separate the Level 2 soil moisture input data production from the actual C3S ECV SM production a second user is available, username c3s_lprm, responsible for the NRT passive input stream. Secure access to the Technical Platform and Operations Framework is exclusively given via SSH using private-key cryptography for authentication. At the moment it is not foreseen to add additional Technical Users.
1.3. C3S Processing Framework
The C3S Processing Framework (CPF) for soil moisture is based upon the scientific base of the ESA CCI Soil Moisture project. This means that the CPF uses the CCI processor of a specific version for merging of active and passive soil moisture products into a merged active, merged passive and combined soil moisture product. Pre and post processing needed to get the input data streams into the correct format to be handled by the CCI processor or to re format the data to the C3S specifications is added around the CCI processor core. Figure 2 shows a high level overview of the complete CPF while Figure 3 provides a more detailed look at the components of the framework.
Figure 2: Simplified overview of the C3S Processing Framework, structured into high level components.
Figure 3: Overview of the elements and interfaces of the C3S Processing Framework
1.3.1. Interface to CCI processor
The close link to the R&D activities in the CCI project is important for bringing new scientific knowledge into C3S. Input and output formats of the CCI processor will stay the same between versions which enables us to replace the CCI processor block (see Figure 2) in the processing framework without the need for large adaption work.
The CCI processor was adapted to run in different modes. In the “reprocessing” mode, default mode, the CCI processor computes processing parameters and produces the most consistent soil moisture CDR based on all available input datasets (historic and current missions). The “NRT” mode of the CCI processor uses these processing parameters to generate the ICDR, taking into account the latest available input data from all current missions, with less computational effort compared to the “reprocessing” mode. Processing parameters are generated during each reprocessing cycle. The parameters are CDF scaling parameters used for scaling the different datasets into the same data space and an error characterization used for weighing different observations. These parameters are stored for each sensor combination and used in “NRT” mode of the processor. Figure 4 shows the scaling parameters and characterized errors for the production of the ICDR in “NRT” mode using the ASCAT, AMSR2 and SMOS sensors. More details about the algorithm and the used parameters can be found in the ATBD [D2].
Figure 4: Overview of the CCI processor in NRT mode
1.3.2. LPRM within C3S Processing Framework
The Land Parameter Retrieval Model (LPRM) is an integral part of the CPF, responsible for the timely production of Level 2 soil moisture products from passive microwave missions. Passive soil moisture products ingested in the C3S ECV SM CPS are produced by VanderSat using the processing facilities provided by the EODC through the C3S project. The historic data from the sensors SMMR, SSM/I, TMI, ASMRE and WindSat are reprocessed when algorithm updates in LPRM are developed. Data from AMSR2 and SMOS are processed in NRT mode.
1.3.3. Post processing
The CCI processor produces daily NetCDF files in both processing modes. Consequently, the C3S post processing block performs the decadal and monthly mean calculation as well as data provisioning to the CDS.
1.3.4. ICDR: Workflow scheduling and monitoring
The production of the ICDR is a time critical task, because the result of each processing run has to be publicly available after a short latency. In order accomplish this requirement the workflow scheduling and monitoring tool Apache Airflow7 is used. It structures the processing chains into directed acyclic graphs (DAG) that can also depend on each other. The processing system is structured into the graphs listed in Table 1 and shown in Figure 5. Apache Airflow automatically triggers each processing run on schedule by taking into account all defined task dependencies. Results of each runs, such as processing time of individual tasks and subtasks runs or processing status, are monitored by Airflow and presented via web-based GUI to the operator.
Table 1: Direct Acyclic Graphs (DAG) defined in the workflow scheduling and monitoring system and their tasks and schedule.
DAG name | Task | Schedule |
amsr2_ingestion | Checks if the AMSR2 files from the LPRM processing chain have arrived. | daily |
amsr2_resampling | Converts the AMSR2 daily images into time series readable by the CCI processor. | dekadal |
ascat_ingestion | Checks if enough PDU files for a day are available. If this is the case then the freeze/thaw retrieval and conversion into time series is performed. | daily |
ascat_resampling | Resampling of ASCAT time series on DGG to CCI target grid. Depends on success of ascat_ingestion. | dekadal |
smos_ingestion | Checks if the SMOS files from LPRM processing chain are available | daily |
smos_resampling | Coverts SMOS files daily images into time series readable by the CCI processor | dekadal |
cci_passive_nrt | Production of the passive C3S product. Depends on amsr2_resampling and smos_resampling. | dekadal |
cci_active_nrt | Production of the active C3S product. Depends on ascat_resampling. | dekadal |
cci_combined_nrt | Production of the combined C3S product. Depends on cci_active_nrt and cci_passive_nrt. | dekadal |
c3s_dekadal_postprocessing | Production of dekadal means and data delivery. | dekadal |
c3s_monthly_postprocessing | Production of monthly means and data delivery. | monthly |
Figure 5: Relationship of the graphs (DAG) in the workflow management system
1.4. Data Management and Streams
Data used in the C3S ECV SM PS can be discriminated into historic and NRT inputs. Historic data are soil moisture products from already decommissioned satellite missions generated in one processing cycle. On the other hand, NRT data are soil moisture products extended and processed on a regular base as soon as new observations from current missions are available. The entire data archive is stored and maintained on the EO Storage system operated by EODC. Raw data (Level 1 and 2) as disseminated by the dataset providers are stored besides the different version of the various processing cycles of the C3S ECV SM PS. The current data archive comprising all C3S relevant datasets is about 21 TB currently including two versions of the CDR and the two of the ICDR. This data archive is available on the disk storage cluster (tier1) of EO Storage and is complemented by regular backups of the data using the third tier the tape library.
For each processing cycle of the TCDR and ICDR, the generated data will be placed in separate directories to assure data integrity of the different versions. Furthermore, data produced by test facilities of the system will be completely separated from the operational data stream for the same reason. The latest version of a soil moisture CDR and ICDR will reside on disk storage (tier1) as long as the data stream is operational. Outdated, older versions of the soil moisture ECV with its corresponding processing parameters are archived in the tape library (tier3).
Movement, deletion of data can only be done by the Operations Team and is closely coordinated with the Science Team.
NRT data streams required for the operational production of the ICDR are regularly monitored by the Operations Team by utilizing workflow routines implemented in Apache Airflow. These checks are executed beforehand of each ICDR production run raising an error if a data stream is broken.
1.4.1. Active input streams
Active input streams are based on observations from a series of C band scatterometers onboard historic and current European satellite missions. Algorithms to derive soil moisture from Cband scatterometers have been developed and are further researched by TU Wien (Wagner et al. 1999a; Wagner et al. 1999b; Wagner et al. 1999c). A production chain of the TU Wien soil moisture algorithms was implemented at EUMETSAT in response of the operational need for numerical weather prediction.
1.4.1.1. ERS AMI-WS
ERS-1 AMI-WS was the first European Cband scatterometer in space launched in 1991. ERS-1 was accompanied by ERS-2 in 1995 after 4 years in space. Together, ERS-1 and ERS-2 comprise a data archive covering more than 20 years of global C-band observations. Soil moisture retrievals from ERS AMI-WS are provided by TU Wien as a research product generated with the Water Retrieval Package (WARP) version 5.5.
1.4.1.2. MetOp ASCAT
MetOp ASCAT is the successor scatterometer mission of ERS AMI-WS supporting operational numerical weather prediction services. H-SAF/EUMETSAT provides NRT products in BUFR format in 3 minute Product Distribution Unit (PDU) files based on the TU Wien soil moisture retrieval. MetOp ASCAT input data streams used for this project are:
- H101 SSM ASCAT-A NRT O: Metop-A ASCAT soil moisture 12.5 km sampling NRT
- H16 SSM ASCAT-B NRT O: Metop-B ASCAT soil moisture 12.5 km sampling NRT
The products are delivered to the EODC by ZAMG and are broadcasted by EUMETSAT via EUMETCast. At the EODC they are taken up by the processing system. The processing consists of the following steps:
- Resampling of the swath based data onto a Discrete Global Grid
- Derivation of freeze/thaw information based on Naeimi [2012]
- Resampling to 0.25 degree CCI/C3S output grid
This results in a time series on a 0.25 degree grid including freeze/thaw information that can be used by the CCI processor.
1.4.2. Passive input streams
In the individual sections for the sensors an overview is given on what specific datasets are currently available for ingestion into the C3S processing system and in case of near-real-time processing, how this is set up. A complete overview of the sensor characteristics, including data availability, product version used, technical specifications, reliability, data quantity and ordering mechanism is described in Chapter 1.1 of D2 the Algorithm Theoretical Basis Document (ATBD). All datasets are stored as NetCDF files in a standardized format.
1.4.2.1. Historical datasets
1.4.2.1.1. SMMR
The SMMR dataset is based on the Nimbus-7 SMMR Level 1B Pathfinder data set, containing brightness temperatures for the period of 1978 to 1987. On the EODC server, currently three datasets are available for the C3S processing system based on SMMR from both the ascending as descending overpasses. First a soil moisture dataset processed based on LPRMv5 algorithm which is currently part of the final soil moisture product (See ATBD), secondly a similar soil moisture dataset processed with the latest algorithm updates, i.e. LPRMv6 (See ATBD), is available which will be implemented in an upcoming TCDR reprocessing cycle. Thirdly a pre processed version of the brightness temperatures are available, which can be directly used to rerun the soil moisture retrieval algorithm in case of algorithmic updates during future TCDR update cycles.
1.4.2.1.2. SSM/I
The SSM/I dataset used within the C3S processing system consists of data from multiple satellites within the Defense Meteorological Satellite Program (DMSP), which are the F08 (1978 to 1991), F11 (1991 to 2000), F13 (1995 to 2009) and the F17 (2006 to present). However, the data is only used and processed for the period before the availability of AMSR-E data in 2002. Both ascending and descending overpasses are processed. Two different dataset are available in the C3S processing system, first the soil moisture dataset processed using LPRMv5 algorithm which is currently part of the C3S soil moisture product, and secondly the preprocessed brightness temperatures, which are rescaled from the EASE grid (25km) to a quarter degree grid and available for future reprocessing activities in the TCDR. No LPRMv6 update is applicable to SSM/I as those updates only apply to the C and X band, and not the Ku band as is used from SSM/I, see ATBD (D2).
1.4.2.1.3. TMI
The TMI data available in the C3S system also consists of three datasets. First the currently used soil moisture dataset based on LPRMv5, secondly an already processed soil moisture dataset based on the latest LPRMv6 algorithmic updates for future implementation during the TCDR reprocessing cycles. Thirdly, also for TMI there is a pre processed brightness temperature dataset available based on TRMM Level 1b calibrated brightness temperatures TRMM TMI, binned into a quarter degree grid. TMI data is available between 1997 and 2015 and only observations between 7pm and 7am (e.g. night time) are used.
1.4.2.1.4. AMSR-E
There are two datasets readily available based on the AMSR-E sensor for the period of 2002 to 2011. First the soil moisture dataset based on the LPRMv6 algorithm update, which was already applied and implemented within the current final C3S soil moisture dataset. Secondly, the pre processed brightness temperatures, binned into a quarter degree grid, which are available for future reprocessing activities. Only descending (1.30 a.m.) overpasses are used.
1.4.2.1.5. WindSat
The WindSat data available for the C3S system consists of 3 different datasets covering the period of 2003 to 2012. First the currently implemented soil moisture dataset based on LPRMv5, secondly a soil moisture dataset based on the improved LPRMv6 algorithm for implementation in future TCDR reprocessing activities, and thirdly the pre processed brightness temperatures. Even though the satellite is still active, no data after 2012 can be used due to data access restrictions, therefore this is not included in the NRT processor for the ICDR. Both ascending and descending overpasses are processed.
1.4.2.2. Datasets for NRT processing
1.4.2.2.1. SMOS
SMOS is part of the NRT processing system and is included in both the CDR products and the 10 daily ICDR extensions.
The processed SMOS database stored on the EODC server consists of pre processed brightness temperatures files which includes re gridding from EASE2 (25km) grid to a quarter degree grid, an extraction of the different angle bins between 35° and 60°, and the inclusion of the ECMWF temperature information as stored in the auxiliary SMOS files compared to the SMOS L3 brightness temperature files. Secondly, a soil moisture dataset was produced using the LPRMv6 algorithm. Both data from the ascending as the descending overpasses is used.
The daily brightness temperatures files (MIR_CDF3TA and MIR_CDF3TD datasets for ascending and descending overpass) and auxiliary files (AUX_CDFECA and AUX_CDFECD) are downloaded through FTP on a daily basis and processed to soil moisture with 1 day delay.
1.4.2.2.2. AMSR2
AMSR2 data is available from 2012 on and is currently processed in NRT (1 day delay) for the C3S production system as part of the ICDR production. Two datasets are available, which is a pre processed brightness temperature dataset and the soil moisture dataset based on LPRMv6, derived from the descending swaths of AMSR2. The L1R AMSR2 brightness temperatures consisting of single swath files are downloaded to the EODC server on a daily basis and then processed into daily quarter degree brightness temperature files with a 1 day lag.
2. Upgrade cycle implementation procedure
System upgrade cycles mainly concern the Technical Platform and Operations Framework as well as the C3S Processing Framework. These cycles are timely planned and tested before the finale implementation takes place. Implementation upgrades are rolled out to the system during maintenance windows, see section 4.1, in order to assure smooth operations of the ICDR production without interruptions. Beforehand, system upgrades are tested by making use of the provided test facilities of the corresponding CDR production line. Notifications in the direction of users will be kept minimal as long as no direct implication of the upgrade is given.
2.1. Technical Platform and Operations Framework upgrades
Upgrades related to the Technical Platform and Operations Framework are implemented and tested by the Operations Team. The utilised tools for hardware and software provisioning and configuration (see section 1.2.2) are fully exploit for upgrade cycles by taking advantage of the source code driven approaches of these tools. With respect to that, hardware setup and software deployments are stored in GitLab under version control. This allows a parallel, automated setup of a fully operational test facility to investigate the foreseen upgrades before deploying them to the operational service. Finally, upgrades will be integrated the operational service during a maintenance window when all tests in the test facility have been successfully passed.
2.2. C3S Processing Framework upgrades
Processing Framework upgrades are triggered by improved retrieval algorithms and the scientific advancements in the CCI SM processor. All released CCI SM product generated with a specific processor version are verified, validated and relevant algorithm improvements are published in the scientific literature. After a new CCI SM processor version is found suitable its integration into the C3S production system is started.
2.2.1. Verification and Validation of CCI SM products
Verification and validation of the CCI SM product generated with a specific CCI processor version is performed by four independent teams in the framework of the project. Procedures to verify and validate the products are performed on different spatial scales and in comparison to different soil moisture products available. Results of this verification and validation activities are regularly summarised in the Product Validation and Intercomparison Report (PVIR) published on the CCI SM web portal.
2.2.2. Integration of CCI SM product updates into C3S
Product and algorithm updates shall only be integrated into C3S after the verification and validation of the CCI SM products was performed successfully. Any significant algorithm updates shall also be published in scientific journals before they are adapted in the C3S project. Implementations of the algorithm updates will be performed in parallel to the operational service at least two months before a new TCDR production cycle is started. Implementation tests provided by the Science Team to assure the correct implementation of the processor have to be passed successfully in order to be considered as ready for operations.
2.2.3. Communication strategy for product updates
After a successful implementation of the upgraded C3S Processing Framework (CPF), the TCDR production is performed as scheduled in the production cycle based on the new version. The TCDR produced with the updated version of the CPF will be validated in accordance to D1. Furthermore, CPF updates require a phase of parallel operations, for at least three months, of two CDR versions to be disseminated to the users to enable tests and adoption to the new product version. After this parallel operations phase the previous product version is stopped and the new version takes over the operational data stream (see Figure 6). In addition, the full TCDR of the new version is provided to users before the ICDR generation of the older version is stopped. This enables users to adapt their processing chains in time.
Communication about the release of a new product version will be started another three months before the release through the Climate Data Store (CDS).
Figure 6: Upgrade cycle from one version of the CDR to the next. A phase of parallel operation gives the users enough time to adapt to the new version.
2.2.3.1. Test product availability
During the parallel operation phase and transition to a new CPF, the new CDR product version will be disseminated through the CDS to the users for adoption of their processing chains.
3. Procedures for reprocessing CDR's
A CDR consists of a TCDR, which represents the archive of the C3S soil moisture product, and the ICDR which is a consistent extension of the TCDR. The ICDR products are generated every dekad (approx. 10 days) and extend the TCDR of the same version as the ICDR. A new version of the CDR will be produced in the following cases:
- Algorithm updates as described in section 2.
- Processing parameter updates.
- Addition of new sensors using an existing algorithm.
- Change in NRT input products that make a reprocessing necessary.
Generally, the number of versions will be kept to a minimum by pooling the possible changes into yearly releases.
3.1. Processing parameter updates
The processing parameters are regenerated during any CDR reprocessing. An explicit processing parameter update is not necessary if another reason for production of a new version of the CDR exists. In the absence of other changes, the processing parameters will be updated yearly after the release of the most recent CDR version.
3.2. Addition of new sensors
The addition of new sensors triggers the release of a new CDR version.
3.3. Change in input products
A change in input products can happen planned or unplanned. Planned changes include improvements of level 2 soil moisture algorithms either for historical missions or NRT products. Unplanned changes can occur in the ingested NRT products for various reasons and will be handled as described in section 4.2.5
4. System maintenance and system failures
System maintenance and failures may arise across all system components, but the most likely affect the Technical Platform and/or the Data Management and Streams. Hereafter, procedures applied in case of system maintenance and failures are described.
4.1. System Maintenance
The timeliness of the C3S ICDR production and delivery is 10 days. Maintenance of individual system components will be planned carefully by the Operations Team resulting in an estimate of the needed effort. Based on the delivery and production of the products, the Operations Team will have a maintenance window of 5 days to perform the required actions. Within these 5 days of maintenance the system will adopted or at least will be prepared for maintenance in the next open maintenance window. In general, maintenance of the system to be carried out is not expected to take so long as to impede the ability to deliver the products with a delay of 10 days. If that is the case, we will follow the system failure procedures.
4.2. System Failures
Failures may arise abruptly any time. System failures are likely caused by failures of the Technical Platform of the system rather than other components of the system. Nevertheless, procedures to be applied in case of individual failures will be summarised in the following. Notification to users about system components failures will only happen if they are directly affected, such as product delivery issues. This notification of the users will be done via the Climate Data Store (CDS) which holds the user database for notification. The period of notice will be latest two days before products are expected by the users.
4.2.1. Loss of key staff
At a first response key staff can be replaced by qualified colleagues from within the partner organisation. If this is not possible then the staff will be replaced by external candidates, in the meantime tasks may be distributed among partners to ensure a continuity of project progress.
4.2.2. Non-performance or loss of consortium members
The consortium team have a long standing successful professional relationship, having worked together over various projects for the last 6 years. As such non-performance is not expected. If a consortium member leaves then an adequate replacement will be sought and will be proposed to ECMWF.
4.2.3. Conflict or dispute between team members
All conflicts and disputes will first be addressed and a resolution sought by the project manager. A method for conflict resolution is in place in sub contracts between EODC and the partners.
4.2.4. Cost overruns
Regular monitoring of the system will allow the project manager and Operations Team to spot and mitigate any costs overruns. If cost overruns cannot be controlled within the project, then ECMWF will be informed and an alternate solution be sought.
4.2.5. Input data stream failure
The product relies upon a suite of EO data sets from both active and passive. The loss of a number of input products would only result in a degradation of the quality of the overall product, and not critically stop the production of the CDS product. Only the loss of all input products would result in the inability to generate a product. Nevertheless, users will be informed about the failure via CDS after the first recognition of the issue.
4.2.6. EO Storage failure
Failures related to the EO Storage system are the most critical ones at the moment. Two scenarios of failure may become apparent. First, the disk storage system (tier1) is not accessible, but tier3 the tape library is still available. In that scenario, an additional available disk storage pool used by the SIDP with much less capacity will be used to recall the required data for processing. No action towards the users is needed as long as the product can be delivered in time. In case of a total blackout of the storage system, tier1 and tier3, currently no backup solution is in place. Accordingly, all users will be notified immediately. Missed products will be disseminated on the next possible release cycle of the ICDR.
4.2.7. Processing Facility Failure
Compute resources of the Technical Platform may fail because of damages in the hardware, broken cooling system, etc. In that case, the Operations Team will start to deploy this part of the system on a remote location such as on TU Wien infrastructure to run the required processing there. Users will be informed if a timeliness delivery of the product is critical (2 days before official publication).
4.2.8. Processing Failure
Processing failure may arise after upgrades in the C3S Processing Framework. Such failures are very unlikely, because updates of the framework are well tested beforehand. In case of a failure, the update will be dropped and the previous version of the C3S Processing Framework will be re-setup. No need to notify the users, because the failure will be resolved immediately and will not affect the user.
5. User support
A dedicated service desk has been set up, the Copernicus User Support (CUS) team, which provides support to users of the CAMS and C3S services at ECMWF. All enquiries about the soil moisture dataset must be submitted through the service desk where it will be dealt with by appropriate persons.
There is a portal (http://copernicus-support.ecmwf.int) where customers can submit enquiries using a form (split into "Data Request", "Documentation and Scientific Questions", "Events, Media and Legal" and "Report an Incident"). The information from this form is passed to the CUS. Once submitted, the user may add comments or further information to the issue, including responding to questions / requests for additional information from the support team.
The C3S 312b Lot4 service provides dedicated level 2 user support to the CUS and is fully registered with the CUS Jira Ticketing Service.
In addition to submitting enquires through the portal a knowledge base is available to users which can be searched for information.
References
Naeimi, V., Paulik, C., Bartsch, A., Wagner, W., Kidd, R., Park, S., Elger, K. (2012). ASCAT Surface State Flag (SSF): Extracting Information on Surface Freeze/Thaw Conditions from Backscatter Data Using an Empirical Threshold-Analysis Algorithm. Geoscience and Remote Sensing, IEEE Transactions on, 50(7), 2566–2582.
Wagner, W., Lemoine, G., Borgeaud, M., & Rott, H. (1999a). A Study of Vegetation Cover Effects on ERS Scatterometer Data. IEEE Transactions on Geoscience and Remote Sensing, 37, 938-948
Wagner, W., Lemoine, G., & Rott, H. (1999b). A Method for Estimating Soil Moisture from ERS Scatterometer and Soil Data. Remote Sensing of Environment, 70, 191-207
Wagner, W., Noll, J., Borgeaud, M., & Rott, H. (1999c). Monitoring soil Moisture over the Canadian Prairies with the ERS Scatterometer. IEEE Trans. Geosci. Rem. Sens., 37, 206-216
This document has been produced in the context of the Copernicus Climate Change Service (C3S).
The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts, operator of C3S on behalf of the European Union (Delegation agreement signed on 11/11/2014). All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose.
The users thereof use the information at their sole risk and liability. For the avoidance of all doubt , the European Commission and the European Centre for Medium - Range Weather Forecasts have no liability in respect of this document, which is merely representing the author's view.
