Contributors: L. Carrea (University of Reading), C. J. Merchant (University of Reading)
Issued by: L Carrea, C Merchant
Date: 31/05/2020
Ref:C3S_312b_Lot4_D1.LK.1-v2.0_202001_System_Quality_Assurance_Document_LSWT_v1.0
Official reference number service contract: 2018/C3S_312b_Lot4_EODC/SC2
History of modifications
List of datasets covered by this document
Related documents
Acronyms
General definitions
L2P – Geophysical variables derived from Level 1 source data on the Level 1 grid (typically the satellite swath projection). Ancillary data and metadata added following GHRSST Data Specification.
L3U – Level 3 Un-collated data are L2 data granules remapped to a regular latitude/longitude grid without combining observations from multiple source files. L3U files will typically be "sparse", corresponding to a single satellite orbit.
L3C – Level 3 Collated data are observations from a single instrument combined into a space-time grid. A typical L3C file may contain all the observations from a single instrument in a 24-hour period.
L3S – Level 3 Super-collated data are observations from more than one satellite that have been gridded together into a single grid-cell estimate, for those periods where more than one satellite data stream delivering the geophysical quantity has been available.
Scope of the document
This document is the System Quality Assurance Document (SQAD) for Lake Surface Water Temperature in the Hydrology service of C3S. It describes contributions to the processing chain, interfaces to C3S, and to RD component, the hardware used in the processing chain, the procedures to implement new data cycles and to reprocess the products, and those applied in case of system failures or anticipated maintenance work, as well as information on user support.
Executive summary
The C3S Lake production system (C3S ECV LK) provides an operational lake surface water temperature (LSWT) and lake water level service, generating lake surface water temperature and lake water level climate datasets for a wide variety of users within the climate change community. Two groups support and maintain the system. One group is dedicated to the lake surface water temperature and the other one to lake water level. This document is about the LSWT system.
The LSWT team is subdivided into a science team, a system development team and an operation team. The science team directs the scientific evolution of the service and provides dedicated scientific support to the users. The system development team is responsible for the implementation of the evolutions, the maintenance and the validation of the processing chain. The operational team is responsible for the Operational Framework with designated responsibilities.
System overview
The C3S Lake production system (C3S ECV LK) is responsible for the operational provisioning of the Lake Essential Climate Variable (ECV) for the Copernicus Climate Change Service. The mission of the system is the timely production of a consistent, long-term, multi-mission and multi-sensor global lake surface water temperature and lake water level data set complying with the defined key performance indicators (KPIs). The two parameters, lake surface water temperature and lake water level, have different systems (Technical Platforms or "processing chains") as their operation requires wholly different input data, ancillary data and algorithms. However, the systems concepts are similar.
Integral components of the overall system for C3S Lake ECV LSWT are:
- the Organizational Framework consisting of
- the Lake Surface Water Temperature (LSWT) group subdivided into roles:
- the LSWT science team
- the LSWT system development team
- the LSWT operation team
- the Lake Surface Water Temperature (LSWT) group subdivided into roles:
- the Technical Platform for LSWT and its Operations Framework
- the C3S Processing Framework (CPF), and
- Data Management and Streams
These components, their functions and interfaces within the C3S ECV LK production system will be discussed in the following.
Organizational Framework
Two groups are cooperating within the organizational framework in order to fulfil the mission objective of the system at the level of the Lake ECV. One oversees the Lake Surface Water Temperature parameter while the other one oversees the Lake Water Level parameter. This SQAD describes the LSWT system.
The Science Team role is to take charge of the scientific validity of the LSWT products generated. The function of the Science Teams is to ensure that the scientific algorithms used are capable of producing a long-term consistent, multi-mission global LSWT products, in accordance with the defined data key performance indicators. The necessary R&D has to be done externally to the project. Previously, R&D was sourced within the project GloboLakes, now completed. It is hoped that future processing cycles will benefit from R&D funded within the auspices of ESA's ongoing Climate Change Initiative, but presently no R&D on the Lakes ECV has been commissioned within that programme. The fundamental scientific approaches are fixed, therefore, and the C3S Science Team focusses on evolutions such as determining parameters necessary to new sensors being brought into the service using existing scientific techniques.
The System Development Team role is to develop and maintain the implementation of the scientific algorithms based on the design of the Science Team and integrate them in the processing baselines. They validate their implementation in accordance with the defined data key performance indicators and produce a new version of the systems that they deliver to the corresponding Operations Team.
The Operations Team role is to deliver the regular production of the LSWT products. With respect to that, the Operations Team undertakes provisioning and configuration of the technical platform as well as the setup and maintenance of the Operations Framework.
These roles are defined separately to help structure the work conceptually. In terms of personnel, the teams overlap considerably.
LSWT Technical Platform and Operations Framework
The Technical Platform and Operations Framework is composed of a set of hardware and software components interacting with each other.
The LSWT Technical Platform is realized at the data and computation facility of the Centre for Environmental Data Analysis (CEDA), here referred to as "JASMIN", and consists of a dedicated project workspace and with shared computational power that are used to generate the test and annual CDRs for LSWT.
JASMIN is a "super-data-cluster", half super-computer and half data-centre, which provides a high performance storage fabric with a co-located batch compute cluster and private cloud. It is aimed primarily at the UK academic research community.
JASMIN provides both "managed" and "unmanaged" cloud infrastructures. The "managed" infrastructure consists of high-performance disk storage alongside compute resources, both for virtualisation (i.e. hosting virtual machines) and batch compute (the LOTUS processing cluster), which together make a managed analysis environment. The disk storage hosts the CEDA Archive as well as "Group Workspaces" which are allocations of high-performance disk shared between members of projects across a wide range of environmental science domains.
In the "managed" part of the infrastructure, STFC-based system administrators are responsible for looking after the systems, installing known and trusted software and managing any outward-facing services, which are made available through the STFC/RAL firewall. These services include the CEDA data access services (FTP, HTTP, OPeNDAP).
Within the managed infrastructure, compute demands are predominantly focused on LOTUS, the batch processing cluster, rather than individual virtual machines. In the "unmanaged" environment, the "Infrastructure as a Service" model of cloud provision is used to provide projects with their own environment in which they can host their own services and workflows. Projects (tenants) are provided with an allocation of storage, CPU, memory and a range of network addresses which form a cloud "virtual organisation" in which they can build their own systems. The tenants provide their own system administrator(s) and maintain the virtual machines themselves, which sit on a portion of network address space assigned to NERC. The tenants manage their own outward-facing services in this area of the network, which sits outside the STFC firewall.
Figure 1 below illustrates the functional view of the JASMIN infrastructure. The C3S SST project is making use of a number of these parts within the project as a whole. For the brokered CDR datasets, the relevant parts are the CEDA archive and download services, shown in green. The ICDR data production further makes use of the CEDA batch compute (LOTUS) and scientific analysis and data transfer machines, as well as group workspace storage. Once short delay processing commences, additional services using the unmanaged cloud, may also be used.
Figure 1: Functional view of the JASMIN infrastructure, showing the range of services provided.
The system described above generates the C3S annual extensions to the LSWT record. The same system was used to generate the brokered dataset from the project GloboLakes, covering the earlier years of the CDR.
The Operations Framework is based on source code that is version-controlled in Github https://github.com/
, holding all code and configurations needed to run the operational service. System operations is done by members of the Operations Team.
C3S LSWT Processing Framework
Figure 2: Processing framework for LSWT
Source data include satellite level 1 (radiance) products from space agencies and numerical weather prediction (NWP) files from ECMWF: the responsibility for collection of these data streams lies with CEDA as part of the agreement of the LSWT service with CEDA. Other auxiliary data are the responsibility of the Science Team (since they embed algorithmic knowledge). The data readers are developed by the System Team in common between LSWT and the C3S SST service, since the data input streams are overlapping, and this ensures consistency.
Processors implement the scientific retrieval methods at full resolution, creating full resolution intermediate outputs, which are compared with validation data. The collection of validation data updates is an annual task of the Science Team (no functional system equivalent to the SST ICOADS system exists for LSWT, so this is done via personal networks).
LSWT data management and Streams
The input data streams for LSWT-4.0 (C3S v1.0 CDR) are listed in Table 1. In future cycles: further streams may be added, particularly including MetOp C AVHRR and the Sentinel 3A and 3B SLSTRs; some data may be sourced from alternative routes (NOAA instead of EUMETSAT); and ERA-Interim will be superseded.
Table 1: Input data streams for LSWT-4.0 (C3S v1.0 CDR)
Satellite data stream | Origin | Responsibility | Collection schedule |
MetOp A & B AVHRR L1 | EUMETSAT | CEDA | Typically 4 months behind |
NWP data stream | Origin | Responsibility | Collection schedule |
ERA Interim | ECMWF | CEDA | Typically 4 months behind |
Validation data stream | Origin | Responsibility | Collection schedule |
Sustained and occasional in situ collections | Network of direct contacts | Science team | Additional data are solicited annually in November/December. |
Upgrade cycle implementation procedure
Technical Platform and Operations Framework upgrades
There is an annual cycle of upgrades to the processing chain and operational framework, linked to the annual production cycle:
- CDR products are generated and disseminated by end of January annually
- Issues to be addressed for the next cycle are identified in part by validation activities and in part by gap analysis, and are diagnosed by April
- Design and initial testing of evolutions culminate in generation of the test CDR product annually by August
- Verification of the test results determines the full implementation of the evolutions to the processing chain and documentation in time for the next CDR generation in January
Scientific evolutions / Processing Framework upgrades
The baseline service for LSWT is to provide consistent LSWT CDRs using the existing algorithms, with ad hoc improvements to those as required implemented as system evolutions. This is because only fundamental R&D will lead to new algorithms and is beyond the scope of the service. Improvements to algorithm parameters and adaptation to newly introduced sensor data streams are in scope for the service, so upgrades will be made, triggered by the availability of revised or additional auxiliary or input data streams.
It is hoped, but is not yet confirmed, that more fundamental R&D for LSWT will be pursued within the ESA Climate Change Initiative. Products and algorithms updates shall only be integrated into C3S after the verification and validation of new CCI LK data/methods are performed successfully. Implementation in the C3S of the algorithm updates will be performed in parallel to the operational service at least two months before a new CDR production cycle is started (i.e., before mid-October). This would trigger a major version number change (in the public-facing versioning nomenclature), to make clear with which brokered data set (GloboLakes LSWT-4.0, or ESA CCI LSWT-5.0, etc) the CDR generated under C3S annually can be seamlessly used.
Communication about the release of a new product version will be started three months prior to release in the Climate Data Store (CDS).
Procedures for reprocessing CDR's (not applicable)
Reprocessing of the long-term LSWT CDR is not expected within the C3S service, so this section is not applicable. Reprocessing is expected within the frame of ESA CCI Lake ECV.
System maintenance and system failures
Any issues or outages, whether planned or unplanned, will be documented in the Quarterly Report.
Dissemination failures
Product dissemination is addressed centrally by EODC for the Hydrology service and is not applicable for this document.
In terms of ensuring EODC can be passed products in a timely fashion, planned outages of the CEDA archive are kept to a minimum, and are publicised in advance to all CEDA users via email, via the CEDA news feed on their website (http://www.ceda.ac.uk/blog/) and Twitter/Facebook channels, enabling planning.
System outages
In the case of unforeseen system failures or other problems affecting data delivery to the C3S the following procedure will be applied:
1. On notification of a problem, internal CEDA procedures will be invoked to investigate and resolve the issue.
2. EODC will be notified by the LSWT Operations Team, who in turn alert the C3S CDS.
It should be noted that support for the CEDA and JASMIN infrastructures and for data delivery issues is only available during standard UK working hours.
System performance
In case of other issues affecting system performance (such as delaying the annual processing), these will be investigated and reported to the CDS as per the case of the system outages above. In the case of complex issues which may take longer to resolve, the progress will be reported regularly to C3S and will be included in the Quarterly Reports.
System failures
Failures may arise abruptly any time, but considering the annual generation of the products, failures can be treated without any major delay in the production, except during the critical generation window (December / January). Notification of EODC and the CDS will be as per the case of the system outages above.
Loss of key staff
At a first response key staff can be replaced by qualified colleagues from within the partner organisation. If this is not possible then the staff will be replaced by external candidates, in the meantime tasks may be distributed among partners to ensure a continuity of project progress.
Input data stream failure
The product relies upon a suite of EO sensors. The loss of one or a few input products will degrade the quality, and not critically stop the production of the CDS product. Only the loss of all input products would result in the inability to generate a product.
User support
A dedicated service desk has been set up, the Copernicus User Support (CUS) team, which provides support to users of the CAMS and C3S services at ECMWF. All enquiries about the LSWT and LWL datasets must be submitted through the service desk where it will be dealt with by appropriate persons.
There is a portal (http://copernicus-support.ecmwf.int) where customers can submit enquiries using a form (split into "Data Request", "Documentation and Scientific Questions", "Events, Media and Legal" and "Report an Incident"). The information from this form is passed to the CUS. Once submitted, the user may add comments or further information to the issue, including responding to questions / requests for additional information from the support team.
The C3S 312b Lot4 service provides dedicated level 2 user support to the CUS and is fully registered with the CUS Jira Ticketing Service.
In addition to submitting enquires through the portal a knowledge base https://confluence.ecmwf.int//display/CKB (resource verified 25-05-20) is available to users which can be searched for information.
